Roshan Kumar is senior product manager at Redis Labs. Metering is not just a simple counting problem. Metering is often confused with measuring, but it is usually more than that. Metering does involve measuring, but as an ongoing process, usually with the goal of regulating the usage or flow of a resource over time. Modern applications incorporate metering in many different ways, ranging from counting people, objects, or events to regulating usage, controlling access, and allocating capacity.Metering solutions generally must process large volumes of data while meeting stringent performance requirements. Depending on the scale of the solution, counting and metering might involve thousands if not millions of updates to a database every second. The primary requirements of a database to support such a solution are high throughput for write operations and low (sub-millisecond) latency for responses.Redis, the open source in-memory database platform, delivers both of these benefits … [Read more...] about How to use Redis for real-time metering applications
It’s taken some time, but Microsoft’s $26 billion purchase of LinkedIn is finally starting to show some interesting results, with LinkedIn data starting to show up in tools like Outlook. It’s the first sign of Microsoft using the social network’s relationship graph, the complex data set that was the reason for one of Microsoft’s biggest Silicon Valley acquisitions.Under the hood, a social network like LinkedIn is nothing more than a huge NoSQL graph database, using a schema-less approach to managing semistructured data. Each node in the graph is an individual, with all his or her profile data. Each node is linked to others, tens or hundreds for people with a few connections, thousands for highly connected individuals. Queries traverse those connections, letting you find all the people you know working on AI, or who are based in Ontario, or who used to work at LinkedIn.[ Working with data in the cloud requires new thinking. InfoWorld shows you the … [Read more...] about Making sense of Microsoft’s graph database strategy
At first glance, building a real-time application may sound like a daunting proposition, one that involves technical challenges as well as a significant financial investment, especially when you have an application goal of responding within a fraction of a second. But advances in hardware, networking, and software—both commercial as well as open source—make building real-time applications today very achievable. So what do these real-time applications look like?This article presents three common real-time application patterns that require a real-time decision, meaning a response returned or transaction executed based on real-time input. To determine which pattern to apply to your application, you must first define your real-time objective. Ask yourself: How fast does the application need to respond?Each application pattern addresses a particular level of real-time response: sub-millisecond, milliseconds, or 100 milliseconds and greater.[ Apache Solr is the hot tool for … [Read more...] about What real-time application pattern works for you?
What is a database? Once upon a time, it was simple. The database was a modern Bob Cratchit putting data in tables made up of very straight columns filled with one row per entry. Long, endless rectangles of information stretching on into the future.The relational database has been the bedrock of modern computing. The vast majority of websites are just a bunch of CSS lipstick painted on top of SQL. Everything that makes us special is just another row in the big table of life.[ Which NoSQL database should you use? Let InfoWorld be your guide. NoSQL standouts: The best key-value databases. | NoSQL standouts: The best document databases. | Keep up with the hottest topics in programming with InfoWorld's App Dev Report newsletter. ]The love affair with the big matrix of bits is slowly fading as developers are realizing that not everything fits into a simple table. And because developers are smart and obsessive about finding solutions for every need, they’ve started creating new and … [Read more...] about 8 technologies transforming the database
If you have lame databases in your on-premises systems, don’t move them to the cloud. They’ll still be lame databases.As thousands of enterprises move their application workloads and data to the cloud, too many move whatever they have, include their lame databases. It’s easy to just lift and shift them you’ll find the popular on-premises databases also available in the cloud. So you end up with the same limitations, just running somewhere new.Don’t do that. Instead, reevaluate the type, and the brand of databases you’re using as part of your cloud migration.[ The RDBMS is scaling out: Review: Google Cloud Spanner takes SQL to NoSQL scale. | MySQL face-off: Amazon Aurora outscales Google Cloud SQL. | Review: ClustrixDB scales out — way out | Review: DeepSQL outruns Amazon Aurora | Keep up with hot topics in programming with InfoWorld's App Dev Report newsletter. ]Use the cloud migration effort to vastly improve your data management and data use … [Read more...] about Fix your databases now as you migrate to the cloud
Roshan Kumar is senior product manager at Redis Labs. Real-time streaming data ingest is a common requirement for many big data use cases. In fields like IoT, e-commerce, security, communications, entertainment, finance, and retail, where so much depends on timely and accurate data-driven decision making, real-time data collection and analysis are in fact core to the business.However, collecting, storing and processing streaming data in large volumes and at high velocity presents architectural challenges. An important first step in delivering real-time data analysis is ensuring that adequate network, compute, storage, and memory resources are available to capture fast data streams. But a company’s software stack must match the performance of its physical infrastructure. Otherwise, businesses will face a massive backlog of data, or worse, missing or incomplete data.[ The InfoWorld review roundup: AWS, Microsoft, Databricks, Google, HPE, and IBM machine learning in the cloud. | Cut … [Read more...] about How to use Redis for real-time stream processing
If you have deployed a few systems of scale, you know that some design problems are worse than others. It’s one thing to write tight code, and another thing to avoid introducing performance-crushing design flaws into the system.Here are nine common problems – poor design choices, really – that will cause your system to spin its wheels, or even turn against itself. Unlike many bad decisions, these can be reversed.[ Database slow? Improve the speed and scalability of your RDBMS with these 21 rules for faster SQL queries. | Keep up with the hottest topics in programming with InfoWorld’s App Dev Report newsletter. ]1. N+1 queriesIf you select all of a customer’s orders in one query then loop through selecting each order’s line items in a query per order, that’s n trips to the database plus one. One big query with an outer join would be more efficient. If you need to pull back fewer at a time you can use a form of paging. Developers using caches … [Read more...] about 9 crushing performance problems in scalable systems
When open source SQL database CrateDB first debuted, its professed mission was to deliver easy, fast analytics on reams of machine-generated data, while running in containerized, cloud-native environments.That mission hasn’t changed with the release of version 2.0, but it has been expanded by way of an enterprise edition with pro-level features. Rather than distribute the enterprise edition as a closed-source, binary blob, the maker of CrateDB is offering it as open source to help speed uptake and participation.[ Also on InfoWorld: 19 open source GitHub projects for security pros. | Track the latest trends in open source with InfoWorld’s Open Source Report newsletter. ]SQL, not slow-QLCrateDB is designed to ingest high-volume, machine-generated data, whether logs from a fleet of servers or sensor data from IoT devices, and make that data accessible through traditional SQL queries. The data may be structured or unstructured; it can be a conventional table, or a freeform … [Read more...] about CrateDB 2.0 Enterprise stresses security and monitoring—and open source
Microsoft apparently missed database godfather Michael Stonebraker’s memo. In 2005 Stonebraker declared the “one size fits all” mentality of the database market is an idea whose “time has come and gone.” Fast forward to 2017 and Microsoft launched Azure Cosmos DB, a new database that promises to do... everything.No, really. Everything.Relational data? Check. Documents? Yep. Graph? Of course. Strong consistency? Bingo! Eventual consistency? That, too! In fact, Cosmos DB has five consistency models to choose from.[ Which NoSQL database should you use? Let InfoWorld be your guide. NoSQL standouts: The best key-value databases. | NoSQL standouts: The best document databases. | Keep up with the hottest topics in programming with InfoWorld's App Dev Report newsletter. ]Not surprisingly, euphoric cries greeted the press release, with one developer gushing that it “absolutely beats any competitor in the cloud” and, as such, “not sure why would … [Read more...] about Does Microsoft’s Cosmos DB promise too much?
MapD, the SQL database and analytics platform that uses GPU acceleration for performance orders of magnitude ahead of CPU-based solutions, has been updated to version 3.0.The update provides a mix of high-end and mundane additions. The high-end goodies consist of deep architectural changes that enable even greater performance gains in clustered environments. But the mundane items are no less important, as they’re aimed at making life easier for enterprise database developers—those most likely to use MapD.[ Download the InfoWorld quick guide: Learn to crunch big data with R. | Sign up for InfoWorld's Big Data Report to stay atop all the latest news and developments in the field. ]Previous versions of MapD (not to be confused with Hadoop/Spark vendor MapR) were able to scale vertically but not horizontally. Users could add more GPUs to a box, but they couldn’t scale MapD across multiple GPU-equipped servers. An online demo shows version 3 allowing … [Read more...] about SQL-powered MapD 3.0 woos enterprise developers
Cloud computing isn’t like working on-premises. Instead of limiting code to one or maybe two datacenters, we’re designing systems that span not just continents but the entire world.And that’s where we start to get issues. Even using fiber connections, the latency of crossing the Atlantic Ocean is around 60ms, though in practice delays are around 75ms. The Pacific is wider, so latency through trans-Pacific fiber is around 100ms.[ NoSQL grudge match: MongoDB and Couchbase Server go nose to nose. | Keep up with hot topics in programming with InfoWorld’s Application Development newsletter. ]Delays add up, and they make it hard to ensure that distributed databases are in sync. That makes it harder still to be sure that a query in the U.K. will return the same result as one in the U.S. Yes, most replication strategies mean that eventually the two will have the same content, but there’s a big question over just when that will happen. If the connections are busy, … [Read more...] about How Cosmos DB ensures data consistency in the global cloud
Microsoft has fired a shot heard around the globe, so to speak, in data management with the debut of Azure Cosmos DB at the recent Microsoft Build 2017 developer conference in Seattle. The cloud database is positioned for elasticity and globally available data, supported on the Azure cloud. The project was founded in 2010 by Microsoft’s Dharma Shukla, who holds the title of distinguished engineer at the company.InfoWorld Editor at Large Paul Krill spoke with Shukla during the conference to get his perspectives on the technology.[ Why Microsoft's Cosmos DB represents the future of cloud databases. | Download InfoWorld's quick guide and get started with Azure Mobile Services for building apps today. | Stay up on the cloud with InfoWorld's Cloud Computing newsletter. ]InfoWorld: Why is this project, which began more than six years ago, going to the public now?Shukla: It’s a very complex system. The goal we had was to build a globally distributed database system, which makes … [Read more...] about Q&A: Microsoft Cosmos DB creator lays out vision for planet-scale database
At first glance, Microsoft's new Cosmos DB Azure database seems like a rebadged successor to Azure's planet-scale NoSQL offering, DocumentDB. It's easy to read Cosmos DB as a point-revision version of its predecessor, down to the fact that existing DocumentDB users will be automigrated.But what's most important about Cosmos DB is not where it's coming from, but where it's heading—and how it may be taking a sizable slice of the cloud-native database world with it. Here are four reasons why Cosmos DB is a harbinger of what's to come for cloud-native database technology and how it's a sign of what's already arrived.[ The cloud storage security gap — and how to close it. | The InfoWorld Deep Dive: How to make document sharing really work in Office 365. ]1. Every major cloud vendor will need to complete with similar optionsHere, "similar" means a single database as a service that offers familiar database metaphors (such as SQL), high consistency and availability, horizontal … [Read more...] about Why Microsoft’s Cosmos DB represents the future of cloud databases
CockroachDB, an open source, fault-tolerant SQL database with horizontal scaling and strong consistency across nodes—and a name few people will likely forget—is now officially available.Cockroach Labs, the company behind its development, touts CockroachDB as a “cloud native” database solution—a system engineered to run as a distributed resource. Version 1.0 is available in both basic and for-pay editions, and both boast features that will appeal to enterprises.[ NoSQL grudge match: MongoDB and Couchbase Server go nose to nose. | Keep up with hot topics in programming with InfoWorld’s Application Development newsletter. ]The company is rolling the dice with its handling of the enterprise edition by also making those components open source and trusting that enterprises will pay for what they use in production.Highly available, highly distributableMany of CockroachDB’s feature were inspired by the work underpinning Google Cloud Spanner, but … [Read more...] about Open source SQL database CockroachDB hits 1.0
MySQL is a bit of an attention hog. With relational databases supposedly put on deathwatch by NoSQL, MySQL should have been edging gracefully to the exit by now (or not so gracefully, like IBM's DB2).Instead, MySQL remains neck-and-neck with Oracle in the database popularity contest, despite nearly two decades less time in the market. More impressive still, while Oracle's popularity keeps falling, MySQL is holding steady. Why?[ MySQL face-off: Amazon vs. Google. | Learn how to get started, step by step, with MySQL. | Also on InfoWorld: 10 essential performance tips for MySQL | Track the latest trends in open source with InfoWorld's Linux Report newsletter. ]An open gift that keeps on givingWhile both MySQL and Oracle lost favor relative to their database peers, as measured by DB-Engines, MySQL remains hugely popular, second only to Oracle (and not by much): DB-Engines Looking at how these two database giants are trending and adding in Microsoft SQL Server, only MySQL continues to … [Read more...] about NoSQL, no problem: Why MySQL is still king
MapD, creator of a GPU-accelerated database that scales both up and out, has open-sourced its core technology.As announced in a press release and blog post, the core database and its "associated visualization libraries" are available under the Apache 2.0 license. But enterprise-level features like the high availability, LDAP, ODBC, and horizontal scaling functionality—many of which debuted in the 3.0 version released earlier this month—will be kept close to the chest.[ Intel, Apache, Amazon, and more: See the 2017 open source rookies of the year. | Cut to the key news in technology trends and IT breakthroughs with the InfoWorld Daily newsletter, our summary of the top tech happenings. ]Core concernsOf the key pieces being open-sourced, the first and most crucial is the MapD Core Database, since it includes the basic bits needed to perform SQL processing on however many GPUs are available from a single server."We wanted the community to be able to take advantage of our core … [Read more...] about MapD’s GPU-powered database is now open source
Earlier this year, Google offered a peek at Cloud Spanner, an automanaged database service that melds features from both conventional relational systems and NoSQL technologies.Today, Google announced Cloud Spanner will be available to the general public later this month. It will compete not only with rival cloud databases, but also up-and-coming open source projects that address scale and reliability issues by using Google's own ideas.[ Get started: A developer’s guide to serverless computing. | Keep up with hot topics in programming with InfoWorld's App Dev Report newsletter. ]The best of both worldsGoogle presents Cloud Spanner as a happy medium between two common database needs that often prove incompatible. A database can be highly scalable and distributed (the NoSQL approach), or it can be transactionally consistent (the conventional database approach). Cloud Spanner aims to be both.As laid out in a 2012 research paper, one key to accomplish this is a time synchronization … [Read more...] about Google’s Cloud Spanner melds transactional consistency, NoSQL scale
Over the past few decades, most IT shops have followed a somewhat similar trajectory: Starting from a centralized model (i.e., the mainframe days), computing resources, much like the cosmological Big Bang, have exploded outwards to become ever-more-distributed and decentralized. This makes sense given market dynamics. Computing platforms evolve quickly, so monolithic computing platforms that require heavy up-front investment are less efficient from a depreciation standpoint (i.e., from a MIPS per dollar per year point of view) than numerous, incremental investments in lower-powered devices. By virtue of ever-more decentralized processing, it logically follows that storage would be (in general) decentralized as well. In fact, storage becomes a balancing act. Data is placed in such a way as to be centralized enough to be manageable, while still being distributed enough to be efficiently used by consumers of that data. That's the paradigm of recent history. But this paradigm is changing … [Read more...] about Is InfoSec Ready for Big Data?
Jean-Marc Robin, CEO of startup Vesalis from Clermont-Ferrand, France, got into the beauty industry, he says, "because I love women." (That phrase in quotes should be read with a French accent if at all possible.) He seems somewhat surprised to find his company's facial recognition technology, built to help department stores sell makeup, drawing interest from governments, French and otherwise. But he does love that they love the technology and its possibilities. Talking with Robin during his visit to Palo Alto, Calif., last week, he definitely seemed like a man tugged in two directions. While he was happy to talk about the successes of the technology in security tests, he kept bringing the conversation back to its applications in department stores, guiding women to selections of hair color and makeup. The company, started in 2005, has eight patents for its facial recognition technology. While it mainly had in mind department store kiosks, where shoppers use it to virtually test … [Read more...] about From Testing Lipstick to Spotting Terrorists
Don't bring the data to your computation if you can help it. Bring your computation to your data.Microsoft has heeded this cardinal rule of information science with the latest Community Technology Preview release of SQL Server 2017. Python can now be used within SQL Server to perform analytics, run machine learning models, or handle most any kind of data-powered work.[ Unleash the power of SQL with 17 tips for faster queries. Download the PDF today! | Also on InfoWorld: 7 essential SQL Server security tips. | Keep up with hot topics in databases with InfoWorld's Big Data newsletter. ]This integration isn't limited to enterprise editions of SQL Server 2017, either—it'll also be available in the free-to-use Express edition.The most conventional application of Python with SQL Server is to execute Python scripts as normal, with SQL Server as a data source. Microsoft has also made it possible to embed Python code directly in SQL Server databases by including the code as a T-SQL stored … [Read more...] about Python power comes to SQL Server 2017