Building custom internal tools with Python and Superblocks

Superblocks is a programmable IDE for developers to build any internal app, workflow, or scheduled job at a fraction of the time and cost. For Python developers, many libraries are included by default; however, there are going to be times when you want to push the boundaries. The Superblocks’ Agent allows developers to extend functionality by importing any packages that are required.

Python data transformation

The most common use of Python that we see is using it to clean up and transform data loaded from a database or API integration before passing it back to an Application. This can be done natively in Python with for loops over the data; however, libraries are there to help. The Pandas library is a great choice for a beginner to delve into and there are many tutorials out there, first load the data into a Dataframe, and then manipulate it to your needs. Finally, return the data as JSON. When doing so, remember to orient by `records`, which converts each row of a DataFrame into a dictionary where each column becomes a key-value pair. This format can be used directly within Superblocks’ tables and charts.

ETL with Pandas

ETL stands for Extract, Transform, Load, which is a process used to collect data from various sources, transform it into a format that can be analyzed and stored in a data warehouse or data lake for later use. It involves three steps:

• Extract: Gathering data from various sources such as databases, files, or APIs.
• Transform: Cleaning, filtering, and converting the data into a consistent format to make it usable for analysis.
• Load: Storing the transformed data in the target data store, such as a data warehouse or data lake.

Superblocks makes this tremendously easy for developers. Extract data with integrations such as Google Sheets, S3 and Snowflake. Transform data with Python and Pandas, before loading into your data stores such as Postgres, MSSQL, and MySQL. Create your ETL as a Superblocks Scheduled Job to have your data processed every night, hour or any time that suits your workflow.

‍

To help with monitoring, push logs to your favorite observability platform such as Datadog or Splunk, or receive updates in Slack or Email should any Job end in a failure.

Importing Python libraries

Superblocks supports common Python and JavaScript libraries in backend APIs. The most popular Python libraries include Numpy, Pandas, Boto3, and Matplotlib, while the most commonly used Javascript libraries are aws-sdk, Base64url, and Moment. By running the Agent, you can also install your own custom libraries from public and private package registries or by using an install script. These are easily added and updated by creating a custom Docker image for the Superblocks agent. The Agent can be run locally for rapid development and via a number of supported deployment methods for running in production. To add image processing and text extraction, opencv2 and tesseract python libraries can be easily imported.

‍

A Workflow is an internal tool that executes a set of steps across your business systems: APIs, Databases, and business logic in code. In this case, a workflow can be created to perform OCR on any image provided to it.

In order to be reusable, this was created as a Workflow rather than an API directly within an Application, enabling it to be called from external custom code, webhooks, or Superblocks Applications, Workflows, or Scheduled Jobs. For example, it can be called from within an Application to load vehicle and owner details for insurance claims processing.

Charting with Plotly

Superblocks is a great choice of platform to visualize your data. Out of the box there is support for quick and easy charting with Line, Bar, Column, Area, Scatter, and Pie charts. However, Superblocks also provides native support for Plotly charts such as `import plotly.express as px`, just be sure to `return fig` at the end of your python step rather than `fig.show()`. Plotly's Python graphing library makes interactive, publication-quality graphs. Examples include area charts, bar charts, error bars, box plots, histograms, heatmaps, subplots, multiple-axes, polar charts and bubble charts. Using a Python step, you can import the Plotly library to bring beautiful and complex charts to your applications.

Predicting Customer Churn with Machine Learning in Python

The Scikit-learn package is also included by default within Superblocks. This brings machine learning capabilities to the platform. There are a number of algorithms that are well suited to predicting customer churn. Just load in your Salesforce data and let Scikit-learn guide your Sales and Customer Success teams toward customers showing indicators of possible churn.

Sharing Applications and Workflows with RBAC and SSO

Applications and Workflows can be shared quickly and easily with individuals or groups via the Share modal for granular role-based access control. Superblocks will take care of all of the headaches of authentication and authorization for you. Combined with native integration into your SSO provider, this will ensure that all data and applications that have been processed with Python are available to only those who need it.

Audit logging

Search the logs of every user action in your internal Applications, Workflows, and Scheduled Jobs execution history. Use Audit Logs to quickly identify errors as well as who is running queries against your data. It's a great tool for troubleshooting any API errors, especially for Scheduled Jobs/Workflows that run during off-hours.

Try out the extensibility of Superblocks with Python

Superblocks has helped thousands of developers save 100s of hours of time both in building and in using internal applications.

‍

To learn more about how to create internal Applications, Jobs, and Workflows with Superblocks view our documentation. Try creating your first application today with a free 14-day trial​​.