In the last decade or so, Open Source software (besides SaaS) has caused a major upheaval in the world of software, both consumer and enterprise. The main attraction of open source software has been the fact that it provides a choice for organizations free themselves from the clasp of the proprietary software vendors and the control they exert in terms of expensive support contracts. Community based and third party support from organizations has also played a major part in the wider adoption of open source software.

Although the essence of Open Source was to make software available for FREE, each software offering does come with a license. As with any free, open standards process, the flavors of Open Source software licenses are aplenty. Before adoption a Open Source offering and benefit from the value delivered  to their organization, it is critical that companies pay close attention and rationalize the nuances of the type of license that governs it. Pay special attention to the fact that each open source technology might itself be an amalgamation of other open source technologies that come with their own licenses.

Given that they are easily available and free and not requiring you to go through a corporate procurement/budget process does not mean companies can start using them willy-nilly. A well-defined governance process is essential for companies to effectively manage the legal and operational risks they could run into down the road.

As part of working with a startup that built software heavily based on open source, I came up with process (which is still being refined). Depending on what stage you are as a company, a brand new startup or a large global organization, you might want to start with different activity in this process, to quickly get a handle on the process.

Inventory

If you are an organization, that is already using some (or many) open source software in your company, and until now, have not had a defined process, the first thing to do is to get a exhaustive inventory of all the open source technologies that are being used.

As part of compiling the inventory, make it specific to the version of the software. There have been many instances where open source license terms have changed between versions of the software. So it is important to capture that.

Once you have a comprehensive inventory created, classify or tag them with the following factors

• license type (Mozilla, BSD, Apache, GPL2…etc)
• License Agreement language (link or downloaded format)
• # usage – number of projects/products using them in the company
• Form of usage (reference or derivations made)
• packaging & distribution – bundled into the product or referenced as a separate pre-requisite
• platforms – (J2EE, PHP, Ruby-on-Rails, .NET)
• Internal Sponsor/lead (identify one, if the decision was made-by-committee or the lead is no longer with the organization)

Approval Policies

Implement an approval process at the outset for approving the decision to adopt any new open source software. In fact, as part of instituting the process, I would strongly recommend, conducting an quick approval process for software already being used as a means to vet the effectiveness of the policy. This gives you a good understanding of what, if any, corrective actions need to be taken to mitigate risks, for software already in use.

An effective approval policy must include roles for all the constituents that will be impacted by adoption of the open source technology. Legal, IT operations, support, engineering, marketing should all have varying degrees of involvement in the approval.

The approval processes should include approvals from

• Legal – for the specific language in the license agreement. This should take into consideration the pattern of usage – reference v creating derivative work, bundling vs soft pre-requisite. If you are selling your product that includes Open Source software, then reviewing your liability clauses to ensure you are covered might also be required.
• IT Operations – in terms of certification for interoperability with remaining technology infrastructure, scalability, security.
• Support – in terms of capability to support your customers while your product, including the open source software being used. This might also mean that you have upgrade to a enterprise support license from your hitherto used community license of the open source software.
• Intellectual Property – clearance in terms of keeping ownership of your intellectual property. Remember, in the event of any M&A your IP will come under scrutiny so a stitch-in-time saves nine. (Startups need to pay special attention to this).

Provisioning

Once the adoption of a open source technology has been approved (legally and operationally), it is important that provisioning is done through proper channels. Much as people hate going through IT for technology provisioning, this one might come back to bite.  Provisioning should involve

• IT certifying the product you plan to use is certified with your current IT infrastructure
• Certification should also be done in other areas like support for security holes, performance/scalability.
• Clear definition of download/uptake process for patches and software. Better yet, have a internal download site for approved patches that are also certified.

Tracking and Governance

In addition to the pre-approval and provisioning, a continuous oversight is necessary in terms of usage of the technology. During the time of usage of the open source technology any of the following could happen

• With the changing technology landscape, companies getting acquired or going bankrupt, you might have inherited risks that you thought you were clear off.
• A motivated developer, seeing new features in the newer version might have upgraded the open source version and built that fancy feature you are planning for the next release.
• Since it is already an approved product, a new project inside the company might start using it and you might not know about it.
• A project using open source technology gets shelved resulting in non-usage of the open source software you

Managing the inventory of the Open Source software being used through some form of Asset Tracking is critical. This will provide a  good platform for period audits against the Terms of Use of each of those software products along with approved projects/users.

In what amounts to a first of its kind – A US bank Merrick sued Savvis following a data breach in 2006 at the bank’s payment processor Card Systems. The claim by the bank was that despite the Cardholder Information Security Program (CISP) certification by Savvis three months prior, unencrypted credit numbers was stolen and bank had to pay fines to the tune of $16M dollars. If this lawsuit stands and Savvis is made accountable for this lapse, this sets a new precedent for future IT Security certifications (or their shortcomings).

In a financial world, auditors have too many shields to protect themselves from liability even after admitting negligence. In case of Arthur Andersen in the Enron scandal, it was held liable for the obstruction justice by shredding related documents and not accounting mistakes or lack of audit oversight (even that too was subsequently overturned if my memory serves me right). That is the limit of my jurisprudence. It is a slightly different story in the personal income tax filing area. The Tax preparer can be held liable as IRS explains.  In IT world, it is a little less clear cut. Technology can be really disparate and complicated and might be difficult to ascertain in some cases as to what it does.

That said, here are some good things that came come out this. Unlike Financial Accounting where both sides (companies and auditors) have qualified individuals that can interpret the law and cannot claim ignorance, in IT world most companies,  instead of wrestling with the vagarities of the mandates, entrust auditors who claim expertise in the area. Companies that can afford an internal auditor might not be able to claim innocence.

With companies increasingly moving their technology portfolio into co-located data centers hosted by third party it creates interesting scenarios when it comes to accountability. Especially with SaaS, it becomes even more interesting. Vendors take ownership of managing and governing customer data and assure them with appropriate certifications like SAS-70. The SaaS solutions themselves might be based on Cloud based infrastructure services like the Amazon Elastic Cloud (EC2). In such cases it creates a chain of claimed certifications that companies and eventually customers have to rely on. Conversely, in the event of a lapse, it could also create a chain of claims customer against the Solution provider who in turn might make a claim against the infrastructure provider. So who will be held accountable?  I will leave that to the experts of law to decide and identify the negligent party.

So what is a company to do faced with this challenge ?

Here are some suggestions

• If you are hiring a auditor for a audit, pay special attention to the liability clauses in the contract. Even if the aforementioned case’ verdict goes either way, this has set a precedent for future audit commitments. Auditors will (and might already be) drafting in a clause to eliminate their culpability in the event of a lapse. But your interest should be to make sure, there are reasonable grounds for elimination of liability. If in doubt err on the side of having an additional, third party certification of the process and the coverage of the audit. I am sure you agree it is cheaper to do the preventative measure than have to pay after a breach.
• As a matter of risk mitigation, don’t leave it to auditors to confirm everything is fine. A certification is just a point in time verification and affirmation of known facts. You need to work at it to keep the certification valid. So have elaborate tools and processes in place – certification or otherwise. Check my earlier post SaaS Data Security – Should I be concerned? for some ideas on mitigating these risks. The concepts apply to an internal IT organization as well.
• Given the gray areas surrounding the compliance mandates (even auditors will attest to that fact) it is important that there is collaboration amongst companies and also vendors on issues that they are confronted with. There are communities like the BreachCenter that are focused on the topics like Data Breaches. Collaborating, participating in such initiatives will evolve best practices that the entire industry can benefit from.

Would love to hear any ideas that companies that are dealing with PCI, SOX, HIPAA are doing to mitigate risks beyond the customary audits. Shoot me an email at smallya at prudentcloud dot com. Would be happy to discuss.

The first mention of SaaS application, as a possible technology choice, is sure to make your IT and the CFO/Risk officer sit up and take interest. A single breach and the consequential data loss can cost companies millions of dollars in penalties/damages. This does not include the unquantifiable damage to the company’s reputation. Regulatory mandates like Sarbanes Oxley (SOX)-404, HIPAA and PCI-DSS have strict requirements on how customer, financial, employee, partner data should be governed and protected. Moving to a SaaS application does not preclude, you the company, from those responsibilities.

Given that, how does a company considering a SaaS application conduct a good assessment of the risks involved before jumping in ?

Let us start with the premise.

Companies store data in servers and databases each kept from unauthorized users under strict access control. Additionally, the data itself is regulated by who can see what and what, if any, operations can they perform on the data. The operations could be manual or via an application that manages it.

In SaaS, your data will reside in the databases and servers owned by the service provider. If your SaaS vendor happens to use third party cloud based services then your data might reside in the data center of the Cloud provider. You as the customer, get to add, update, delete data from within the SaaS application, subject to the business rules and security policies implemented in the application. Unlike in the case of an on-premise application, your IT organization will not retain access to the servers, databases, storage, backups and the network. That responsibility would now rest with the service provider.

Risk Mitigation

In order to safeguard your data that would reside in the service provider’s database, here are somethings, you must ask the software vendor as part of the RFP/evaluation process.

1. Keep the bad guys away: Knowledge of existence of a particular service and its location is not a secret. Everyone knows how to access Salesforce.com or Netsuite. You go to the vendor’s site and look for the Customer Login or Client Login button/tab. Given that what are some of the processes vendor has in place for preventing Denial of Service Attacks, Spoofing (remember the Salesforce.com incident!).
2. Authentication/Sign-On: Most SaaS vendors these days support and delegate Sign-On using SAML(Security Assertion Markup Language). This will allow you to configure the entry point to the application/data for your company through a trusted site – like your enterprise portal which is accessed through VPN. With such a configuration you are now essentially in-charge of your provisioning and revocation of access from your corporate  single sign-on identity management.
3. Encryption Policies: Making data secure in the data center is the first step. Another challenge is to make sure data is safe in-transit. As you access data from the application, data is traveling over the wire back and forth. Having strong encryption of data on the wire is paramount. 128bit SSL encryption is common these days, but some vendors are now starting to provide stronger encryption. Check what your vendor supports. In fact, while you are at it also check what they support for the on-disk encryption so your data in storage and backups are encrypted.
4. Test the Tester:Verify the quality process being used to conduct security tests. Specifically check for tests conducted to identify vulnerabilities due to Cross-site Scripting, Cookie Management, Mass Update of Access Control, iFrame embedding, URL manipulation, Excessive Logging.
5. Multi-tenancy/Data Slicing: Multi-tenancy provides the economies of scale that SaaS vendors seek to provide low subscription costs. But this also means your data will be co-mingled with other customers. With all the rapid product development cycles, if the tenancy data separation architecture is not robust, this might expose your data to your competitors. So it is important to understand the way data separation is implemented.  Have your architects verify the  architecture to understand the multi-tenancy architecture better. Specifically check for the quality tests conducted to prevent SQL Injection. Code flaws that allow SQL Injection would end up allowing access to wrong slice of data.
6. Network Security: Network weakness is one of common ways for malicious users to get access to information. Typical issues found in networks would be improper SSL configuration, lack of robust session management and open ports. Once the hacker gets access, they can hijack active sessions and gain access to user credentials and critical information.
7. Backup/Recovery: In the quest for 99.99% availability, it is conceivable that vendors build redundancy and replicate data just in case of a crash. This means copies of your data could be residing in multiple data centers, in some case in multiple geographies. So if you are in a regulated industry and comply to data security guidelines that prevent your data being hosted outside the country or a certain geography, you should get that clarified upfront.
8. Certification: First of, ask for a SAS-70 Type II audit certificate, preferably conducted in the last 6 months and as an ongoing practice, every 6 months. This is like an insurance policy. SAS-70 is a generalist guideline and it is not a mandate. The certification, by itself, does not guarantee that everything is hunky-dory. You can find many examples of cases where companies supposedly certified, having incidents of data breaches resulting in large financial losses. If the application you are using include managing Credit Cards or Health Care data then you should also ask for the specific certification like PCI-DSS and HIPAA.
9. Preventive Measures: As part of your evaluation process, request for the documented architecture and policies for Intrusion Detection System (IDS) and Intrusion Prevention Systems (IPS). In addition, also ask for a recent run report of the IDS system. Review this with your corporate security team and ensure they meet your corporate mandate. If you are a small business and do not have a corporate IT security team, hire a CISA certified consultant and review the report with them. Even better, insist on a penetration test to be conducted by your team. You can hire third party services or a third party software to conduct a round of penetration testing.
10. Audit: Request for a report from a Penetration Test conducted recently, preferably by a third party. This is like conducting a fire drill to verify the preventive and corrective process the vendor has in place, does work.  More about Penetration Testing and Intrusion Detection and how to incorporate this into your process, later in the post.
11. Governance: Request for a change management and access management report of who in the vendor’s organization has access to the data. If the vendor has SAS-70 Type II certification, this is something they would have documented already. With the dynamic environment, in which most SaaS vendors operate, there will be a lot of churn in the people. It is important to make sure the vendor has processes in place to ensure their past employees, contractors do not retain access after leaving the organization.
12. Understand the data management:To provide 99.9%, almost interrupted, high performance service levels, SaaS vendors will end up replicating your data (or backing up) to multiple data centers. It is important to understand that process and access control on the replicated data.

This should give you a good set of upfront checks before you decide on a SaaS vendor. But just like physical fitness, security is not a one time thing. It is a ongoing process. You keep at it regularly – Measure, Monitor and Adapt, and only then can you be sure you data is secure.

Portability/Switching Cost

One of the beauties of SaaS is that if the SaaS vendor measure up to their commitments in SLA, you have the opportunity to switch. No infrastructure, resource investment overhangs. But don’t expect for switching to be as easy as switching your cellphone service. You still have the all important data residing with the vendor. With a little bit of smarts during initial contract negotiation, might get you your data free or for a nominal cost, you still have to ensure that your data is deleted clean from the vendor’s database and servers after you leave. A breach at your previous vendor and learning that your old data was part of the data loss is not something you would want to hear.

Bake it into the Contract

To make this a IT priority and a scheduled activity, here are terms you should incorporate into your Contract.

1. Have your vendor furnish a SAS-70 Type II certificate every 6 months or a year (depending on your comfort level)
2. Conduct a penetration testing exercise every 4-6 months from your end. If you are happy with the third party agency employed by the vendor to conduct a penetration test then save yourself some money and ask for that report to be made available to you. Vendors like Qualys provide you with a service that you can avail for conducting these tests.
3. Have your vendor furnish IDS/IPS logs to be available upon request or through the Self-Service Administration portal.

Parting Shot

You know I am big SaaS fan, so now for you SaaS naysayers out there – chew on this.

If it makes you feel any better, these are the very same checks and processes that your internal IT has to follow. So not going with SaaS does not preclude you from this process. With SaaS, since this is asked of the vendor and goes through the scrutiny of many customers like you, the chances are their process would be much more hardened resulting in your data being more safer. As un-comforting as it is the last I checked, the majority of data thefts happened from the inside of an enterprise as this survey done by a UK firm states – 33% of employees would steal data .

Why would someone need a copy of production instance?

Some of the most common reasons are

• to create a test environment with representative production data
• to create a production support environment
• to create a custom development environment.
• to do volume testing
• to perform integration testing
• and many more

What should we be asking ourselves?

Every time you are faced with this need, besides the entire logistics, the hardware, software and storage needs, you have think about the governance aspects. (If not asked, it better be)

1. How do we make sure critical/personal data stored in production is not exposed to the unauthorized users of the development environment?
2. What subset of data should we prune?
3. Which data do I mask/scramble?  If, I did that how would itmpact the quality of the environment?

What is critical data?

If you consider the entire ERP, Supply Chain and CRM product footprint, there are large number of data points that are considered business critical, personal and legal. Any/All such data points are considered out of bounds when it comes to access by any of the unauthorized users. The user base we need to think about are third party consultants, testers, IT staff, helpdesk, partners, you get the idea.

Here are some of the key entities/data elements that you MUST have in your list of things to scramble

1. HR Data (employee SSN, Date of Birth, Addresses, phone numbers)
2. Payroll Information (employee payroll data, bank information)
3. User Login Information (encrypted password in FND_USER)
4. Credit Card Information (stored for either Receivables or Payables)
5. Supplier Bank Information (in case it is stored for automated payments)
6. Customer Information (contacts, addresses, bank accounts, if any)
7. Critical Sales Opportunities (could include material information deemed as insider information)

What is Data Scrambling?

Data Scrambling or Data Masking is a technique used to mask critical data sets, attributes so the critical data is not visible to the users of the cloned/non-production database copied from production. Steven Chan has a detailed explanation on the same.  The Application Management Pack in Oracle Application Manager allows administrators to define policies to scramble data on the clone.

You should also check out the Data Sheet Oracle has published on Data Masking.

Plan of action

Any of those replicated/cloned environments are considered open or semi-regulated and hence the pre-requisite for such environments should be data masking, pruning to eliminate any of the critical data.

Work with your corporate counsel to understand the regulatory compliance mandates that you are required to comply with. If you are a public company, most often that not, you will be required to comply with Sarbanes Oxley (SOX) 103,105,404 and 802  and PCI, Graham Billey Leach Act. If you are in the Health Care industry you might have needs to comply with HIPAA and likewise if you are in a Process Manufacturing company you might have CFR Part 11 and other regulations.

All the guidelines that apply on a production database in terms of access control/data security also apply to any copies of the production database.

If you are looking for more detailed information or help with defining Data Scrambling policies send me a note. I will be happy to share.

Audit is becoming the norm in most companies. Thanx to all the myriad regulatory requirements, SOX, HIPAA, PCI DS, Gramm-Leach-Biley Act (GLBA), California State, Japan SOX, IT Organizations across the world are spending enormous amounts of time and money in meeting the requirements.

In my discussion with a IT Manager at a billion dollar company with substantial sized IT Operations he was referring to the unrealistic set of demands he gets from his internal auditor. While some of them make sense, some of them were just outright ridiculous. While not trivializing the value of audits, he related an incident that happened at his company where a purchasing buyer was caught procuring personal stuff on the company account and having it delivered at work. He felt that with good audit capabilities in place, she would not have been able to continue with her illegal activities.

The problem in most cases is that there is no clear definition of what constitutes compliance and what is not. In the absence of clear mandates, auditors (internal and external) are just trying to cover all the bases as information director for security at Sony alludes to.

Onerous as it is, the benefits of implementing the compliance mandated controls are not challenged. With the huge array of applications, systems deployed in a company, the IT Organizations most often have scant idea of what is out there in the haystack. With the governance being forced via these mandates, the companies are getting to track and keep tabs on all the goings-on in their IT universe. Here is starter list of things for you to be well-prepared for an audit

• Application Access Controls: As a rule of thumb, any application should have ways to control what a user can see, do and when. The first thing any auditors would ask is for a report which outlines the access control. Special focus will be on the privileged users of systems (read your DBAs, System Administrators). You should not be surprised if you get  request to curtail the privileges of  “Super” users with unlimited privileges.

Oracle E-Business Suite, as most of you know, comes with “SYSADMIN” responsibilities which are akin to the “keys-to-the-kingdom”. A user with this responsibility can grant herself any other responsibilities, change system wide settings and pretty much do everything. Due to the inherent design/architecture of the application this becomes a necessary evil. To address this limitation, one of the key things to do is to audit all the activities performed by a user assuming that responsibility. A report of Privileged User Activity is definitely something you should be monitoring. Another important thing that keeps coming up is the audit on the provisioning of user responsibilities.

• Database Access Controls: Database access control is the most critical area to monitor. It goes without saying that someone with database access can do a lot of harm if they are not wired the right way. So regulating the access to the database and audit all the actions performed when connected is paramount. Oracle database allows you to control who has access database can be by restricting it to some IP address, known users, known applications. In this case, the most common mandate that comes from audit side is to grant access to only the must-have tables and not all of them, must-have users and not a free for all read only access.
• Password Policies: Auditors would be very interested in knowing the corporate password policies relating to frequency of changing them, complexity of passwords. If  you know Oracle E-Business Suite you know, the password management is not one of their strong suit. It is not all that tough to decrypt a password in Oracle E-Business Suite. In most customer instances, this is addressed by delegating the credential management to their corporate identity management. Another key area of concern that auditors raise is around the cloning of production instance and the carry over of production passwords to the cloned test/dev instance. PCI and HIPAA compliance mandates require you to scramble data as part of replication of critical data. Have your operational policies around cloning, data scrambling ready for the auditors.
• Log Policies: Policies around managing log files would also be of interest to auditors.  Log files are stores for critical information about servers, ports, account names and with inadequate error handling,  error stacks could be divulging critical information. Log file locations, access control on those directories, purge policies, log file content etc all come into the discussion. Having some kind of log miner to constantly monitor contents of the log is one of areas of investments companies have made in recent years to tackle this challenge.
• Change Management Controls and Logs: Application Change Management is of lot of interest to the auditors. Here are some of the things you should certainly expect to be asked.
  • Change Request Work flow and Approvals
  • Impact and Risk Analysis process
  • Backout process
  • Review and Audit logs of the changes once their completed.
  • Exceptions made to the process, reasons and approvals for the same
  • Segregation of Duties – clear delineation of roles in the change lifecycle i.e., Requesters, Creators, Approvers, Implementer, Reviewer and Auditor
• Patch Management: Another critical thing that gets reviewed during audit is Patch Management process. Automation of the same would make life easier. Any manual patching would undergo scrutiny around access control, log management, reviews post implementation are guaranteed questions that come up.
• Software Updates: One of the other things that constantly comes up but does not have clear answer is  if the system up to date on patches. While you should exercise discretion and do your due diligence in applying all the patches the vendor provides. But at a minimum an analysis has to be done and a decision had to have been made with sufficient substantiation.

If you have not been part of an audit so far, and have now started wondering, if this will mean a lot of reporting, you guessed it right. Audits have become part of life and companies that have invested in automation have stood to gain.
What are your thoughts? Shoot me an email if you think of anything I missed.