In human behavior analysis, the Visual Focus Of Attention (VFOA) of a person is a very important cue. VFOA detection is difficult, though, especially in a unconstrained and crowded environment, typical of video surveillance scenarios. We estimate the VFOA by defining the Subjective View Frustum, which approximates the visual field of a person in a 2D (ground floor) representation of the scene. This opens up to several intriguing behavioral investigations. In particular, we propose the Inter-Relation Pattern Matrix, that suggests possible social interactions between the people.