Visibility polygon

Naive algorithms are easy to understand and implement, but they are not optimal, since they can be much slower than other algorithms.

In real life, a glowing point illuminates the region visible to it because it emits light in every direction. This can be simulated by shooting rays in many directions. Suppose that the point is ${\displaystyle p}$ and the set of obstacles is ${\displaystyle S}$. Then, the pseudocode may be expressed in the following way:

algorithm naive_bad_algorithm(${\displaystyle p}$, ${\displaystyle S}$) is
    ${\displaystyle V}$ := ${\displaystyle \emptyset }$
    for ${\displaystyle \theta =0,\cdots ,2\pi }$:
        // shoot a ray with angle ${\displaystyle \theta }$
        ${\displaystyle r}$ := ${\displaystyle \infty }$
        for each obstacle in ${\displaystyle S}$:
            ${\displaystyle r}$ := min(${\displaystyle r}$, distance from ${\displaystyle p}$ to the obstacle in this direction)
        add vertex ${\displaystyle (\theta ,r)}$ to ${\displaystyle V}$
    return ${\displaystyle V}$

Now, if it were possible to try all the infinitely many angles, the result would be correct. Unfortunately, it is impossible to really try every single direction due to the limitations of computers. An approximation can be created by casting many, say, 50 rays spaced uniformly apart. However, this is not an exact solution, since small obstacles might be missed by two adjacent rays entirely. Furthermore, it is very slow, since one may have to shoot many rays to gain a roughly similar solution, and the output visibility polygon may have many more vertices in it than necessary.

The previously described algorithm can be significantly improved in both speed and correctness by making the observation that it suffices to only shoot rays to every obstacle's vertices. This is because any bends or corners along the boundary of a visibility polygon must be due to some corner (i.e. a vertex) in an obstacle.

algorithm naive_better_algorithm(${\displaystyle p}$, ${\displaystyle S}$) is
    ${\displaystyle V}$ := ${\displaystyle \emptyset }$
    for each obstacle ${\displaystyle b}$ in ${\displaystyle S}$:
        for each vertex ${\displaystyle v}$ of ${\displaystyle b}$:
            // shoot a ray from ${\displaystyle p}$ to ${\displaystyle v}$
            ${\displaystyle r}$ := distance from ${\displaystyle p}$ to ${\displaystyle v}$
            ${\displaystyle \theta }$ := angle of ${\displaystyle v}$ with respect to ${\displaystyle p}$
            for each obstacle ${\displaystyle b'}$ in ${\displaystyle S}$:
                ${\displaystyle r}$ := min(${\displaystyle r}$, distance from ${\displaystyle p}$ to ${\displaystyle b'}$)
            add vertex ${\displaystyle (\theta ,r)}$ to ${\displaystyle V}$
    return ${\displaystyle V}$

The above algorithm may not be correct. See the discussion under Talk.

The time complexity of this algorithm is ${\displaystyle O(n^{2})}$. This is because the algorithm shoots a ray to every one of the ${\displaystyle n}$ vertices, and to check where the ray ends, it has to check for intersection with every one of the ${\displaystyle O(n)}$ obstacles. This is sufficient for many simple applications such as video games, and as such many online tutorials teach this method.[8] However, as we shall see later, there are faster ${\displaystyle O(n\log n)}$ algorithms (or even faster ones if the obstacle is a simple polygon or if there are a fixed number of polygonal holes).

A visibility polygon for a point in the center (shown in white) inside a simple polygon, outlined in black.

Given a simple polygon ${\displaystyle {\mathcal {P}}}$ and a point ${\displaystyle p}$, a linear time algorithm is optimal for computing the region in ${\displaystyle {\mathcal {P}}}$ that is visible from ${\displaystyle p}$. Such an algorithm was first proposed in 1981.[2] However, it is quite complicated. In 1983, a conceptually simpler algorithm was proposed,[3] which had a minor error that was corrected in 1987.[4]

The latter algorithm will be briefly explained here. It simply walks around the boundary of the polygon ${\displaystyle {\mathcal {P}}}$, processing the vertices in the order in which they appear, while maintaining a stack of vertices ${\displaystyle {\mathcal {S}}=s_{0},s_{1},\cdots ,s_{t}}$ where ${\displaystyle s_{t}}$ is the top of the stack. The stack constitutes the vertices encountered so far which are visible to ${\displaystyle p}$. If, later during the execution of the algorithm, some new vertices are encountered that obscure part of ${\displaystyle {\mathcal {S}}}$, then the obscured vertices in ${\displaystyle {\mathcal {S}}}$ will be popped from the stack. By the time the algorithm terminates, ${\displaystyle {\mathcal {S}}}$ will consist of all the visible vertices, i.e. the desired visibility polygon. An efficient implementation was published in 2014.[9]

For a point in a polygon with ${\displaystyle h}$ holes and ${\displaystyle n}$ vertices in total, it can be shown that in the worst case, a ${\displaystyle \Theta (n+h\log h)}$ algorithm is optimal. Such an algorithm was proposed in 1995 together with its proof of optimality.[10] However, it relies on the linear time polygon triangulation algorithm by Chazelle, which is extremely complex.

A visibility polygon for a point in the center (shown in white) amongst a set of arbitrary line segments in the plane, allowed to intersect only at their endpoints, acting as obstacles (shown in black).

For a point among a set of ${\displaystyle n}$ segments that do not intersect except at their endpoints, it can be shown that in the worst case, a ${\displaystyle \Theta (n\log n)}$ algorithm is optimal. This is because a visibility polygon algorithm must output the vertices of the visibility polygon in sorted order, hence the problem of sorting can be reduced to computing a visibility polygon.[11]

Notice that any algorithm that computes a visibility polygon for a point among segments can be used to compute a visibility polygon for all other kinds of polygonal obstacles, since any polygon can be decomposed into segments.

A divide-and-conquer algorithm to compute the visibility polygon was proposed in 1987.[12]

An angular sweep, i.e. rotational plane sweep algorithm to compute the visibility polygon was proposed in 1985[13] and 1986.[11] The idea is to first sort all the segment endpoints by polar angle, and then iterate over them in that order. During the iteration, the event line is maintained as a heap. An efficient implementation was published in 2014.[9]

For a point among generally intersecting segments, the visibility polygon problem is reducible, in linear time, to the lower envelope problem. By the Davenport–Schinzel argument, the lower envelope in the worst case has ${\displaystyle \Theta (n\alpha (n))}$ number of vertices, where ${\displaystyle \alpha (n)}$ is the inverse Ackermann function. A worst case optimal divide-and-conquer algorithm running in ${\displaystyle \Theta (n\log n)}$ time was created by John Hershberger in 1989.[14]

• VisiLibity: A free open source C++ library for visibility computations in planar polygonal environments.
• visibility-polygon-js: A public domain Javascript library for computing a visibility polygon for a point among segments using the angular sweep method.