Use various data source types to quickly generate text data for artificial datasets.

In a previous article, we explored creating many-to-one relationships between columns in an artificial PySpark DataFrame. This DataFrame only consisted of Foreign Key information and we didn’t produce any textual information that is perhaps useful in a demo DataSet.

For anyone seeking to populate a man-made dataset, it is probably going it would be best to produce descriptive data similar to product information, location details, customer demographics, etc.

On this post, we’ll dig into a number of sources that might be used to create synthetic text data at little effort and value, and use the techniques to drag together a DataFrame containing customer details.

Synthetic datasets are a fantastic approach to anonymously show your data product, similar to an internet site or analytics platform. Allowing users and stakeholders to interact with example data, exposing meaningful evaluation without breaching any privacy concerns with sensitive data.

It could actually even be great for exploring Machine Learning algorithms, allowing Data Scientists to coach models within the case of limited real data.

Performance testing Data Engineering pipeline activities is one other great use case for synthetic data, giving teams the flexibility to ramp up the dimensions of knowledge pushed through an infrastructure and discover weaknesses within the design, in addition to benchmarking runtimes.

In my case, I’m currently creating an example dataset to performance-test some Power BI capabilities at high volumes, which I’ll be writing about in the end.

The dataset will contain sales data, including transaction amounts and other descriptive features similar to store location, worker name and customer email address.

Taking off easy, we will use some built-in functionality to generate random text data. Importing the random and string Python modules, we will use the next easy function to create a random string of the specified length.