Paranoid? Masking, anonymizing, and obfuscating PII in XML and JSON data

Amar Manipatra Flexter

Data masking, anonymization, and obfuscation are methods to scramble personally identifiable information (PII). We can apply different techniques to mask data, e.g. you may substitute values (swap na for another) or randomly substitute characters just preserving the data type.

There are no open source tools (at least we are not aware of any) that automate masking of data in XML or JSON. That has changed as of today. We are releasing Paranoid an open source tool to mask XML and JSON data.

Pypi link - https://pypi.org/project/PARANOID/

Bitbucket - https://bitbucket.org/sonra/paranoid/src/master/

Paranoid and Flexter

Paranoid is best used in combination with Flexter.

Flexter, is an enterprise XML conversion tool. It converts XML and JSON to text, a relational database or Hadoop/Spark (ORC, Parquet, Avro). We provide a free trial version of Flexter online. If you have privacy concerns around uploading your data you can now download and use Paranoid to mask individual elements.

Paranoid is developed in Python. You will need Python 2.6+ to use it. You can either mask all of the elements in an individual XML/JSON, a folder of XML/JSON, or just individual elements. Paranoid replaces sensitive data with random characters. It can’t be decoded.

Here is a list of the Paranoid switches:

If you are still concerned with privacy you can always check the script yourself (Remember it is an open source tool after all).

Paranoid masking in action

In this blog we will show you how Paranoid and Flexter work hand in hand to mask and convert data in XML.

Anonymising all elements in a single XML document

In a first step we will mask an individual XML element. Let’s look at a sample XML file before masking.

Let’s use Paranoid in a next step to mask all of the XML elements in this sample document.

We can see the argument -I given for our input file sample.xml and argument -o for the output location.

Now let’s look at the head of the xml file before and after masking. All of the XML elements (apart from namespaces) have been obfuscated with random characters.

Original File(SAMPLE)(Complete files you will find below)

Mask File(SAMPLE)

Masking all elements in multiple JSON/XML documents

Well all this looks pretty cool, but what if you want all the files in a directory/sub directory to be masked? What if the directory contains both JSON and XML?

Not to worry. Paranoid can also handle this requirement.

Masking large XML files

This is all great. But what if you have a large file and less memory?

Well Paranoid has a solution for this as well. It offers a switch -b where you can provide a byte size and Paranoid will read the specified size mentioned by you at a moment and mask those contents at the point. Let’s demonstrate it.

That’s right paranoid just buffered 2000 bytes and streamed the data to the output location.

Masking individual tags in an XML document

There is one more thing we wanted to show you. You can be selective in the tags you mask.

You can provide a comma separated list of XPaths to mask.

We are masking two tags: download_date and url

Original File

Mask File

Using Flexter on masked and original data

For data analytics it is typically better to have data in a relational form. Querying data in XML is hard or near impossible. SQL is the lingua franca for data. Flexter is the perfect tool to convert your masked data to a relational format. With the free version you can convert XML and JSON files to text. The premium version also supports conversion of files to a database or popular Hadoop formats, e.g. ORC, Parquet, Avro. You can find out more about Flexter in our product and FAQ section. We are also working on a pay per use SaaS version of Flexter. This version comes with an API. Please refer to our blog post Converting clinical trials XML data to Snowflake and Quicksight with Flexter. We are looking for beta testers. Please reach out if you are interested.

Let’s have a look at the output of Flexter before masking

Here is the output of Flexter after masking. Random characters