Data Scraping

What is data scraping?

Data scraping, in its most general form, refers to a technique in which a computer program extracts data from output generated from another program. Data scraping is commonly manifest in web scraping, the process of using an application to extract valuable information from a website.

https://www.cloudflare.com/en-in/learning/bots/what-is-data-scraping/

How Data Scraping Is Done 

 Web scraping is a fairly direct process when viewed at a high level. Code is utilized to pull information, generally via a scraper bot. The bot sends a request to the website, parses the HTML document and converts it into a different format.

Over time, the game has grown more sophisticated. As scraper bots become successful, content protection strategies are beefed up to thwart their efforts. In turn, the bots respond by developing tactics to outmaneuver these new protection mechanisms — and so it goes.   

For the scrapers, content may be derived at little or no expense. Instead of having to write their own content, conduct research and obtain customer reviews, for example, the scrapers may post material on their sites. They avoid having to pay for certain reports and other documents.  

Two Types of Data Scraping

Web Scraping

If you’ve ever copy and pasted information from a website, you’ve performed the same function as any web scraper, only on a microscopic, manual scale.

Web scraping, also known as web data extraction, is the process of retrieving or “scraping” data from a website. Unlike the mundane, mind-numbing process of manually extracting data, web scraping uses intelligent automation to retrieve hundreds, millions, or even billions of data points from the internet’s seemingly endless frontier.

Screen Scraping

Screen scraping is the act of copying information that shows on a digital display so it can be used for another purpose. Visual data can be collected as raw text from on-screen elements such as a text or images that appear on the desktop, in an application or on a website. Screen scraping can be performed automatically with a scraping program or manually with an individual extracting data.

How is web scraping stopped completely?

The only way to totally stop web scraping is to avoid putting content on a website entirely. However, using an advanced bot management solution can help websites eliminate access for scraper bots almost completely.

What is the difference between data scraping and data crawling?

Crawling refers to the process large search engines like Google undertake when they send their robot crawlers, such as Googlebot, out into the network to index Internet content. Scraping, on the other hand, is typically structured specifically to extract data from a particular website.

Sources / References:

https://www.datamation.com/big-data/data-scraping/

https://www.cloudflare.com/en-in/learning/bots/what-is-data-scraping/

Digital twin (DT)

 Abstract on Digital twin (DT):

Digital twin (DT) is one of the most promising enabling technologies for realizing smart manufacturing and Industry 4.0. DTs are characterized by the seamless integration between the cyber and physical spaces. The importance of DTs is increasingly recognized by both academia and industry. It has been almost 15 years since the concept of the DT was initially proposed. To date, many DT applications have been successfully implemented in different industries, including product design, production, prognostics and health management, and some other fields. However, at present, no paper has focused on the review of DT applications in industry. In an effort to understand the development and application of DTs in industry, this paper thoroughly reviews the state-of-the-art of the DT research concerning the key components of DTs, the current development of DTs, and the major DT applications in industry. This paper also outlines the current challenges and some possible directions for future work.

What is a digital twin?

A digital twin is a digital representation of a physical object or system. The technology behind digital twins has expanded to include large items such as buildings, factories and even cities, and some have said people and processes can have digital twins, expanding the concept even further. The idea first arose at NASA: full-scale mockups of early space capsules, used on the ground to mirror and diagnose problems in orbit, eventually gave way to fully digital simulations.

But the term really took off after Gartner named digital twins as one of its top 10 strategic technology trends for 2017 saying that within three to five years, “billions of things will be represented by digital twins, a dynamic software model of a physical thing or system".  A year later, Gartner once again named digital twins as a top trend, saying that “with an estimated 21 billion connected sensors and endpoints by 2020, digital twins will exist for billions of things in the near future."

In essence, a digital twin is a computer program that takes real-world data about a physical object or system as inputs and produces as outputs predications or simulations of how that physical object or system will be affected by those inputs.

Why and How to Design Digital Twins?

As mentioned above, digital twins can be created for a wide range of applications, for example, to test a prototype or design, assess how a product or process will work under different conditions, and determine and monitor lifecycles.

A digital twin design is made by gathering data and creating computational models to test it. This can include an interface between the digital model and an actual physical object to send and receive feedback and data in real time.

Data

A digital twin requires data about an object or process in order for a virtual model to be created that can represent the behaviours or states of the real world item or procedure. This data may relate to the lifecycle of a product and include design specifications, production processes or engineering information. It can also include production information including equipment, materials, parts, methods and quality control. Data can also be related to operation, such as real-time feedback, historical analysis and maintenance records. Other data used in digital twin design can include business data or end-of-life procedures.

Modelling

Once the data has been gathered it can be used to create computational analytical models to show operating effects, predict states such as fatigue, and determine behaviours. These models can prescribe actions based on engineering simulations, physics, chemistry, statistics, machine learning, artificial intelligence, business logic or objectives. These models can be displayed via 3D representations and augmented reality modelling in order to aid human understanding of the findings.

Linking

The findings from digital twins can be linked to create an overview, such as by taking the findings of equipment twins and putting them into a production line twin, which can then inform a factory-scale digital twin. By using linked digital twins in this way it is possible to enable smart industrial applications for real world operational developments and improvements.

Where is it Used?

Digital twins are used in a wide variety of industries for a range of applications and purposes. Some notable examples include:

Manufacture

Automotive

Retail

Healthcare

Disaster Management

Smart Cities

References: 

https://ieeexplore.ieee.org/document/8477101

https://www.networkworld.com/article/3280225/what-is-digital-twin-technology-and-why-it-matters.html

https://en.wikipedia.org/wiki/Digital_twin

https://www.twi-global.com/technical-knowledge/faqs/what-is-digital-twin

Microsoft Hololens

 Abstract

Seminar on Hololens is Microsoft’s take on augmented reality, which they call “mixed reality”. Using multiple sensors, advanced optics, and holographic processing that melds seamlessly with its environment, These holograms can be used to display information, blend with the real world, or even simulate a virtual world. 

Microsoft HoloLens, known under development as Project Baraboo, are a pair of mixed reality smartglasses developed and manufactured by Microsoft. HoloLens was the first head-mounted display running the Windows Mixed Reality platform under the Windows 10 computer operating system. The tracking technology used in HoloLens can trace its lineage to Kinect, an add-on for Microsoft's Xbox game console that was introduced in 2010

What Is HoloLens?

HoloLens is an untethered, fully self-contained Windows 10 computer that rests comfortably on your head. It’s what’s known as a mixed reality device, a device that tries to blend the real and digital worlds. You see objects placed in the world that look and—to an extent—act like they’re in the real world. In contrast, VR immerses you in an environment and you typically don’t see anything around you but that virtual world. You generally aren’t visually aware of the real world outside your head-mounted display (HMD).  This experience can take you flying in outer space while you sit in your office chair. And AR tries to enhance the world around you with extra data, such as markers, or heads-up information that may pertain to your location. Some AR headsets simply throw text and images on a screen overlapping whatever you’re looking at.

With the HoloLens, you can bring applications and objects into the world around you that understand your environment. If you want an application pinned to the wall or in mid-air like a digital screen,  no problem. Such apps stay put, even when you leave your room and come back the next day. I’m constantly leaving virtual windows open in other rooms, to be surprised when I go back days later and they’re still there. And that’s not all. Suppose you want a skeleton standing in front of you in your living room that you can walk around and inspect (including climbing on your couch to look at the top of the head). Again, no problem. Drop a virtual 3D object, say a ball—referred to as a hologram—into your world and it will fall and hit your real table and stop. Move the table and the ball will fall and hit your real floor. The HoloLens understands the world around you and most are absolutely amazed the first time they try it (though I’m still waiting to be able to download Kung Fu into my brain).

How does it work?

The Hololens has a plethora of optical sensors, with two on each side for peripheral “environment understanding” sensing, a main downward facing depth camera to pick up hand motions, and specialized speakers that simulate sound from anywhere in the room. The Hololens also has several microphones, an HD camera, an ambient light sensor, and Microsoft’s custom “Holographic Processing Unit” that they claim has more processing power than the average laptop. All this comes together to sense the spatial orientation of the unit in the room, track walls and objects in the room, and blend holograms into the environment.

Reference:

https://www.gvsu.edu/cms4/asset/7E70FBB5-0BBC-EF4C-A56CBB9121AECA7F/7_things_about_microsoft_hololens.pdf

https://en.wikipedia.org/wiki/Microsoft_HoloLens

https://docs.microsoft.com/en-us/archive/msdn-magazine/2016/november/hololens-introduction-to-the-hololens