Local or Cloud
AI/ML Data Cleansing
Golden Record Pipeline
486 Patterson Ave
Winston-Salem, NC 27101
– – –
11 Park Place
New York, NY, 1007
– – –
Bagmane Laurel, Krishnappa
Garden, C V Raman Nagar,
Karnataka 560093, India
– – –
1644 Platte Street
Denver, CO 80202
– – –
Lange Dreef 11
4131 NJ Vianen
You’re ready to start. You’ve read about data-centric systems or have an idea for a Web3 service and discovered Fluree is the right way to get started. You’ve downloaded Fluree, starred it on Github, and reviewed the docs. Now what?
Should you just start building, or plan your project out first with an architectural diagram? You may be wondering about the risks associated with diving in headfirst: what if the components you have chosen don’t scale easily, or at all?
I’m Jake, a platform engineer at Fluree, and in this post, I’m going to show how easy it is to get started with Fluree and still have confidence that it will grow with you and scale to meet your system requirements.
You can start working with Fluree On-Demand right away. But if you want to work towards your own custom fluree deployment, below are some options for Fluree Anywhere.
Crawl – Getting Started with Fluree Anywhere
Getting started with Fluree Anywhere doesn’t mean you have to architect for distributed computing from the start. You can step into it. You can start with an all-in-one standalone version that includes everything in one package. Even though this is one package, it includes:
Walk – Starting to Grow
As you start to grow your system, you might want to add more query servers in order to scale your ability to handle an increase in demand generated by your users. You might also create a React or NodeJS app to communicate with the ledger. This new layer allows you to control how your users or administrators view the data or create new data.
Run – Scaling Out
Lastly in the run stage, you may need to scale your ledger servers. You might want to build a server cluster to handle losing one or more ledger servers, or you are starting to create multiple ledgers that have their transactions handled by this cluster.
I like to use Docker for all of these deployments. It helps with dependency management and ensures I’m not running something that could conflict with Fluree. This is especially true on my local workstation, maybe less so on a single-purpose server. But if you have JDK ≥ 11 setup, you can use our download and start the ledger server directly.
Below, I’ll show you a simple Crawl → Walk → Run progression for both local and cloud deployments of Fluree-Anywhere. Local would represent how you might provision Fluree in your own data center or server(s), and cloud represents how you may use a third-party to deploy and scale Fluree.
If you have Docker Desktop running already you can run Fluree without installing anything else.
docker run -p 8090:8090 fluree/ledger:v1.0.0-beta8
This will start an ephemeral instance of Fluree and you’ll be able to see the log output to the screen.
Once it is running, you can visit http://localhost:8090 to see the admin UI.
If you go to another terminal window while Fluree is running, you can run docker ps to see the container id. With it you can run:
docker stop [container_id]
docker start [container_id]
I like to create a folder on my host machine so that when I start creating data and want to keep it, I can use it with new Docker containers later on.
docker run -v /Users/username/ledger_data/:/var/lib/fluree -p 8090:8090
Of course, you don’t need Docker. If you have java jdk >= 11, you can follow the instructions and install Fluree from the download.
Once you are comfortable working with the admin UI and are able to create and query data, you’ll probably want to start development on an app to connect to Fluree. You can review the API endpoints here and use a tool like Postman to simulate how your app might consume and transact with Fluree.
If you’re working with React, you could start with our cra-template
If your app is taking on too much load from all of the queries that are taking place, it might be time to add a query server in front of the all-in-one you are running. You can run one or more instances of our nodejs server.
If you need higher availability on the transaction side, you will start thinking about running more instances of the ledger server. Our ledger servers are able to run in a group and use a raft consensus algorithm in order to maintain consistency across nodes.
If you start with 3 nodes, you can lose one and still be operational.
Our Docker Fluree Example repo has a docker-compose.cluster.yml file that displays how this can be done.
If you have docker-compose you can run this directly:
docker-compose -f docker-compose.cluster.yml up
Even if you aren’t using Docker, you can see in the file the environment variables that you can use for each instance of the ledger server to group them together.
Running Fluree in your own cloud has similar steps going from a crawl to walk to run. The cloud is someone else’s servers, after all.
To get started, you can spin up an EC2 instance at AWS or a Droplet at DigitalOcean or wherever you want to launch a VM. From there you can install java jdk >= 11 and download and install Fluree. Or you can install Docker and run the all-in-one version of the ledger server as we did above. Note that Fluree will run on port 8090 unless you specify otherwise, and you’ll need to add that to a public url http://exampleurl.com:8090 to be able to access your running admin UI.
Because Fluree uses web communications through HTTP and Web Sockets, scaling with readily-available web tools is straightforward.
For the walk stage in the cloud, I would put the ledger server behind a load balancer or a proxy to control web traffic to it. You can still run the single instance, but a load balancer will give you added protection against DDOS attacks, etc.
Also, you can add the nodejs query server like I explained above in the local “walk” section and put the ledger server on a private network with traffic only coming from the query server(s).
If you’re requiring even more availability and cannot afford any downtime, like in the local run section, you can create a group of ledger servers and connect them together in a cluster. It would make sense to put some of them in different availability zones to avoid a data center outage.
You’ll also want to put your query servers behind a load balancer that can auto-scale out as far as you’d like.
You don’t have to start with a complicated system to begin building with Fluree. Getting started with the Crawl steps can help you build a solid mental model of how everything works before solidifying your growth plans.
Just know that Fluree can scale with you as your needs demand it.
I’m a fan of infrastructure-as-code and would recommend looking into Terraform, AWS Cdk, or Pulimi once you get past the crawl stage of cloud deployment.
Thanks for taking the time to read more about Fluree!
Follow us on Linkedin
Join our Mailing List
Subscribe to our LinkedIn Newsletter
Subscribe to our YouTube channel
Partner, Analytic Strategy Partners; Frederick H. Rawson Professor in Medicine and Computer Science, University of Chicago and Chief of the Section of Biomedical Data Science in the Department of Medicine
Robert Grossman has been working in the field of data science, machine learning, big data, and distributed computing for over 25 years. He is a faculty member at the University of Chicago, where he is the Jim and Karen Frank Director of the Center for Translational Data Science. He is the Principal Investigator for the Genomic Data Commons, one of the largest collections of harmonized cancer genomics data in the world.
He founded Analytic Strategy Partners in 2016, which helps companies develop analytic strategies, improve their analytic operations, and evaluate potential analytic acquisitions and opportunities. From 2002-2015, he was the Founder and Managing Partner of Open Data Group (now ModelOp), which was one of the pioneers scaling predictive analytics to large datasets and helping companies develop and deploy innovative analytic solutions. From 1996 to 2001, he was the Founder and CEO of Magnify, which is now part of Lexis-Nexis (RELX Group) and provides predictive analytics solutions to the insurance industry.
Robert is also the Chair of the Open Commons Consortium (OCC), which is a not-for-profit that manages and operates cloud computing infrastructure to support scientific, medical, health care and environmental research.
Connect with Robert on Linkedin
Founder, DataStraits Inc., Chief Revenue Officer, 3i Infotech Ltd
Sudeep Nadkarni has decades of experience in scaling managed services and hi-tech product firms. He has driven several new ventures and corporate turnarounds resulting in one IPO and three $1B+ exits. VC/PE firms have entrusted Sudeep with key executive roles that include entering new opportunity areas, leading global sales, scaling operations & post-merger integrations.
Sudeep has broad international experience having worked, lived, and led firms operating in US, UK, Middle East, Asia & Africa. He is passionate about bringing innovative business products to market that leverage web 3.0 technologies and have embedded governance risk and compliance.
Connect with Sudeep on Linkedin
CEO, Data4Real LLC
Julia Bardmesser is a technology, architecture and data strategy executive, board member and advisor. In addition to her role as CEO of Data4Real LLC, she currently serves as Chair of Technology Advisory Council, Women Leaders In Data & AI (WLDA). She is a recognized thought leader in data driven digital transformation with over 30 years of experience in building technology and business capabilities that enable business growth, innovation, and agility. Julia has led transformational initiatives in many financial services companies such as Voya Financial, Deutsche Bank Citi, FINRA, Freddie Mac, and others.
Julia is a much sought-after speaker and mentor in the industry, and she has received recognition across the industry for her significant contributions. She has been named to engatica 2023 list of World’s Top 200 Business and Technology Innovators; received 2022 WLDA Changemaker in AI award; has been named to CDO Magazine’s List of Global Data Power Wdomen three years in the row (2020-2022); named Top 150 Business Transformation Leader by Constellation Research in 2019; and recognized as the Best Data Management Practitioner by A-Team Data Management Insight in 2017.
Connect with Julia on Linkedin
Senior Advisor, Board Member, Strategic Investor
After nine years leading the rescue and turnaround of Banco del Progreso in the Dominican Republic culminating with its acquisition by Scotiabank (for a 2.7x book value multiple), Mark focuses on advisory relationships and Boards of Directors where he brings the breadth of his prior consulting and banking/payments experience.
In 2018, Mark founded Alberdi Advisory Corporation where he is engaged in advisory services for the biotechnology, technology, distribution, and financial services industries. Mark enjoys working with founders of successful businesses as well as start-ups and VC; he serves on several Boards of Directors and Advisory Boards including MPX – Marco Polo Exchange – providing world-class systems and support to interconnect Broker-Dealers and Family Offices around the world and Fluree – focusing on web3 and blockchain. He is actively engaged in strategic advisory with the founder and Executive Committee of the Biotechnology Institute of Spain with over 50 patents and sales of its world-class regenerative therapies in more than 30 countries.
Prior work experience includes leadership positions with MasterCard, IBM/PwC, Kearney, BBVA and Citibank. Mark has worked in over 30 countries – extensively across Europe and the Americas as well as occasional experiences in Asia.
Connect with Mark on Linkedin
Chair of the Board, Enterprise Data Management Council
Peter Serenita was one of the first Chief Data Officers (CDOs) in financial services. He was a 28-year veteran of JPMorgan having held several key positions in business and information technology including the role of Chief Data Officer of the Worldwide Securities division. Subsequently, Peter became HSBC’s first Group Chief Data Officer, focusing on establishing a global data organization and capability to improve data consistency across the firm. More recently, Peter was the Enterprise Chief Data Officer for Scotiabank focused on defining and implementing a data management capability to improve data quality.
Peter is currently the Chairman of the Enterprise Data Management Council, a trade organization advancing data management globally across industries. Peter was a member of the inaugural Financial Research Advisory Committee (under the U.S. Department of Treasury) tasked with improving data quality in regulatory submissions to identify systemic risk.
Connect with Peter on Linkedin
Turn Data Chaos into Data Clarity
"*" indicates required fields
Enter details below to access the whitepaper.
Pawan came to Fluree via its acquisition of ZettaLabs, an AI based data cleansing and mastering company.His previous experiences include IBM where he was part of the Strategy, Business Development and Operations team at IBM Watson Health’s Provider business. Prior to that Pawan spent 10 years with Thomson Reuters in the UK, US, and the Middle East. During his tenure he held executive positions in Finance, Sales and Corporate Development and Strategy. He is an alumnus of The Georgia Institute of Technology and Georgia State University.
Connect with Pawan on Linkedin
Andrew “Flip” Filipowski is one of the world’s most successful high-tech entrepreneurs, philanthropists and industry visionaries. Mr. Filipowski serves as Co-founder and Co-CEO of Fluree, where he seeks to bring trust, security, and versatility to data.
Mr. Filipowski also serves as co-founder, chairman and chief executive officer of SilkRoad Equity, a global private investment firm, as well as the co-founder, of Tally Capital.
Mr. Filipowski was the former COO of Cullinet, the largest software company of the 1980’s. Mr. Filipowski founded and served as Chairman and CEO of PLATINUM technology, where he grew PLATINUM into the 8th largest software company in the world at the time of its sale to Computer Associates for $4 billion – the largest such transaction for a software company at the time. Upside Magazine named Mr. Filipowski one of the Top 100 Most Influential People in Information Technology. A recipient of Entrepreneur of the Year Awards from both Ernst & Young and Merrill Lynch, Mr. Filipowski has also been awarded the Young President’s Organization Legacy Award and the Anti-Defamation League’s Torch of Liberty award for his work fighting hate on the Internet.
Mr. Filipowski is or has been a founder, director or executive of various companies, including: Fuel 50, Veriblock, MissionMode, Onramp Branding, House of Blues, Blue Rhino Littermaid and dozens of other recognized enterprises.
Connect with Flip on Linkedin
Brian is the Co-founder and Co-CEO of Fluree, PBC, a North Carolina-based Public Benefit Corporation.
Platz was an entrepreneur and executive throughout the early internet days and SaaS boom, having founded the popular A-list apart web development community, along with a host of successful SaaS companies. He is now helping companies navigate the complexity of the enterprise data transformation movement.
Previous to establishing Fluree, Brian co-founded SilkRoad Technology which grew to over 2,000 customers and 500 employees in 12 global offices. Brian sits on the board of Fuel50 and Odigia, and is an advisor to Fabric Inc.
Connect with Brian on Linkedin
Eliud Polanco is a seasoned data executive with extensive experience in leading global enterprise data transformation and management initiatives. Previous to his current role as President of Fluree, a data collaboration and transformation company, Eliud was formerly the Head of Analytics at Scotiabank, Global Head of Analytics and Big Data at HSBC, head of Anti-Financial Crime Technology Architecture for U.S.DeutscheBank, and Head of Data Innovation at Citi.
In his most recent role as Head of Analytics and Data Standards at Scotiabank, Eliud led a full-spectrum data transformation initiative to implement new tools and technology architecture strategies, both on-premises as well as on Cloud, for ingesting, analyzing, cleansing, and creating consumption ready data assets.
Connect with Eliud on Linkedin
Get the right data into the right hands.
Build your Verifiable Credentials/DID solution with Fluree.
Wherever you are in your Knowledge Graph journey, Fluree has the tools and technology to unify data based on universal meaning, answer complex questions that span your business, and democratize insights across your organization.
Build real-time data collaboration that spans internal and external organizational boundaries, with protections and controls to meet evolving data policy and privacy regulations.
Fluree Sense auto-discovers data fitting across applications and data lakes, cleans and formats them into JSON-LD, and loads them into Fluree’s trusted data platform for sharing, analytics, and re-use.
Transform legacy data into linked, semantic knowledge graphs. Fluree Sense automates the data mappings from local formats to a universal ontology and transforms the flat files into RDF.
Whether you are consolidating data silos, migrating your data to a new platform, or building an MDM platform, we can help you build clean, accurate, and reliable golden records.
Our enterprise users receive exclusive support and even more features. Book a call with our sales team to get started.
Download Stable Version
Download Pre-Release Version
Register for Alpha Version
By downloading and running Fluree you agree to our terms of service (pdf).
Hello this is some content.