Perfection Series: Forgotten Data in your Logs (Log Redaction Service) by Andy Thurai

From a business standpoint, leaking sensitive information into your logs is not only bad, but could lead to compliance, liability, and risk disaster sooner than you think. While there are solutions, including DLP, out there to inspect the data traffic and help capture sensitive data leakage, not many solutions out there are proactive and intrusive enough inspect the backplane of your systems for sensitive data leakage or regulatory compliance analysis. This becomes more pronounced when you have multiple ways that you allow users (especially the admin users) to access your system – such as browser, command line, XML interface, etc. You need to worry not only about the logs for each of these interfaces, but also the types of logs that are kept and where they may go in the future; i.e. – records such as trace log, transactional log, exception log, command log, admin log. etc.

Recently Intel/McAfee Service Gateway (ESG/MSG) have seen a lot of activity and interest in providing API services in/from the cloud. One of the major issues, faced was that the log, which is stored in the cloud, might contain information that is sensitive or a compliance issue, especially when you offer this as a service (SaaS) and exposed 24×7 to the hackers in the cloud. While this detailed logging may not be an issue for the enterprise customers that keep the actual log information in a centralized secure storage, most times this becomes an issue when you offer a multi-tenant environment and share resource with other users.  In order to provide more control to the customers in the cloud, we introduced a new log redaction feauture, which can be used either in the enterprise version or in the cloud version. This helps customers with sensitive information such as PAN data (especially with credit card information), personal data PII, names, addresses, SS#, DOB and other pertinent information including passwords in verbose modes. Intel’s solution allows that data to be removed/masked with ease and it is user definable, both for patterns and for masking specifics.

I was talking to a customer of mine few days ago on this very topic and I told them how cool this is. He replied, “I think our system is very secure and we have taken ‘extra measures’ given that they deal with multiple compliance standards and issues.” So I suggested to perform a log spider scan. He called me 2 days later panicked with what he found. If you don’t know whether you should worry about this or not there are spider scan tools available on the net, just Google it and see what your logs tell you. If you don’t like what you see don’t blame me :) .

While most customers take extra care of their transactional messages, I have seen a lot of customers a bit lax in regards to logging and administrative interfaces. I recently blogged about an incident with a customer with exposed data in the logs which you can read here.

Our solution allows them to tighten their logs up multiple notches. It comes with about 30 or so pre-defined filters, with an option for customers to build more on their own, using a simple visual tool. It can be applied to any level of logs including the most verbose levels. The masks are user defined and are flexible. Once you turn them on, define them, there is no need to restart your system, it is always on after that until you explicitly turn it off. What is even cooler is that the logs would be instantly cleaned once you push the config, and the push is cluster wide into all node points. Imagine the power of controlling the sensitive log data in all edge devices (whether Enterprise edge or extended to the cloud) in one push of a button.

In reality, the logging system normally logs the content given from many different underlying components, such as from input server, invocation agent, runtime workflow, mediation engine, security engine, etc. This makes it complicated to manage with so many components. For instance, when you log things from the input side (at some verbose log levels, such as detailed trace), it can log the wire data which can be anything (imagine that most solutions out there log everything that comes on the wire for auditing purposes). So it is very hard to prevent the sensitive data from being logged without special handling, contrary to what you may think.

Imagine if you are dealing with PCI, HIPAA, etc and have an edge device to define, saying I need my logs to be cleaned of said sensitive data and define masking/encryption on transactional data as well. You can be sure that from your edge inside, or going out, your message and logs are all cleaned to your satisfaction and for compliance.

If you need more information on this or on our solutions in general please check out or reach out to me.

Andy Thurai — Chief Architect & CTO, Application Security and Identity Products, Intel

Andy Thurai is Chief Architect and CTO of Application Security and Identity Products with Intel, where he is responsible for architecting SOA, Cloud, Governance, Security, and Identity solutions for their major corporate customers. In his role, he is responsible for helping Intel/McAfee field sales, technical teams and customer executives. Prior to this role, he has held technology architecture leadership and executive positions with L-1 Identity Solutions, IBM (Datapower), BMC, CSC, and Nortel. His interests and expertise include Cloud, SOA, identity management, security, governance, and SaaS. He holds a degree in Electrical and Electronics engineering and has over 20+ years of IT experience.

He blogs regularly at on Security, SOA, Identity, Governance and Cloud topics. You can find him on LinkedIn at


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: