Enterprise Visibility Architect

Last month in Security Application Instrumentation I wrote:

Right now you're [developers] being taught (hopefully) "secure coding." I would like to see the next innovation be security application instrumentation, where you devise your application to report not only performance and fault logging, but also security and compliance logging.

This is a forward-looking plea. In the meantime, we are stuck with numerous platforms, operating systems, applications, and data (POAD) for which we have zero visibility.

I suggest that enterprises consider hiring or assigning a new role -- Enterprise Visibility Architect. The role of the EVA is to identify visibility deficiencies in existing and future POAD and design solutions to instrument these resources.

What does this mean in real life? Here is an example. Let's say a company operates a FTP proxy for business use. Consider various stakeholders involved with that server and the sorts of visibility they might want:

• Data center managers want physical accountability for the server. They also want to know how much power is consumed and how much heat is output.
• Network administrators want to know how much bandwidth the server uses for OS and application updates. They also want to know how much bandwidth is used by data transfers and backup processes.
• System administraors want to know if the asset is performing properly.
• Users and asset owners want to know how much data is transferred (in the vent they are billed for this service).
• Human resources administrators and legal want to know what files are transferred, potentially to identify fraud, waste, and abuse.
• Auditors want to validate and asses secure configurations.
• Security analysts want to resist, detect, and respond to incidents.
• Forensic investigators want to know the state of the asset and the files transferred to investigate incidents.

These are all requirements that should be included in the design of the server before it is deployed. However, no one does all of this, and only a few organizations accomplish a few of these items. The role of the EVA is to ensure all of these requirements are built into the design.

When these requirements are not built into the design (as is the case with 95+% of all infrastructure, I would wager) it's the job of the EVA to work with concerned parties to introduce visibility through instrumentation. For example, how could the investigators' concern be met?

• If the proxy supports logging, enable it. (This is usually not done because of "performance concerns." If the resource were appropriately sized prior to deployment to account for visibility, this would not be a problem.)
• Add a passive device to monitor traffic to and from the proxy server. Application-aware monitoring tools like Bro's FTP Analyzer can record FTP control channel activities. (The resistance to this technique involves not wanting to configure a SPAN port, or lack of a SPAN port. I prefer taps, but inserting the tap requires scheduling downtime -- another sticking point.)
• If the investigator only wants IP addresses for the endpoints, then NetFlow could be enabled on a router through which traffic to the FTP server passes. Note that NetFlow cannot be configured to only provide flow data for a specific port (like 21 TCP) so filtering would have to happen on the NetFlow collector. Using NetFlow effectively requires building a NetFlow collector. (Other concerns include loading the router, which could have been accounted for when the design for this business system was created.)
• If the investigator only wants IP addresses for the endpoints, then a logging ACL could be enabled on a router through which traffic to the FTP server passes. Hits on this ACL could be exported via Syslog. Using Syslog requires building a Syslog server. (This option will also add load to the router.)
• Depending on the architecture, intervening firewalls could also be configured to log connection details in the same manner that NetFlow or router ACLs do.

I believe that logging integrated into the application (i.e., the FTP process) is the best option when one is designing a new resource. When visibility is introduced after the asset is deployed, instrumenting it becomes more difficult.

If you hadn't guessed, I am becoming the de facto EVA in my job as director of incident response because I need data to detect and respond to incidents. However, all of the stakeholders are natural allies because they want to know more about various assets.

Thanks to the I want to believe generator for the image above.Copyright 2003-2007 Richard Bejtlich