| NSF Org: |
CNS Division Of Computer and Network Systems |
| Recipient: |
|
| Initial Amendment Date: | August 15, 2019 |
| Latest Amendment Date: | August 15, 2019 |
| Award Number: | 1910085 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Deepankar Medhi
dmedhi@nsf.gov (703)292-2935 CNS Division Of Computer and Network Systems CSE Direct For Computer & Info Scie & Enginr |
| Start Date: | August 15, 2019 |
| End Date: | July 31, 2022 (Estimated) |
| Total Intended Award Amount: | $214,874.00 |
| Total Awarded Amount to Date: | $214,874.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
3 RUTGERS PLZ NEW BRUNSWICK NJ US 08901-8559 (848)932-0150 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
NJ US 08854-3925 |
| Primary Place of Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): |
CSR-Computer Systems Research, Networking Technology and Syst |
| Primary Program Source: |
|
| Program Reference Code(s): |
|
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.070 |
ABSTRACT
Virtual reality (VR) video streaming has been gaining popularity recently with the rapid adoption of mobile head mounted display (HMD) devices in the consumer video market. As the cost for the immersive experience drops, VR video streaming introduces new bandwidth and performance challenges, especially in live streaming, due to the delivery of 360-degree views. This project develops a new content-based viewport prediction framework to improve the bandwidth and performance in live VR streaming, which predicts the user's viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user's viewport of interest.
This project consists of three research thrusts. First, it develops a content-based viewport prediction framework for live VR streaming by tracking the motions and semantics of the objects. Second, it employs hardware and software techniques to facilitate real-time execution and scale the viewport prediction mechanism to a large number of users. Third, it develops evaluation frameworks to verify the functionality, performance, and scalability of the approach. The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter.
With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization. Moreover, the interdisciplinary nature of the project will enhance the education and recruitment of underrepresented minorities in several science, technology, engineering, and mathematics (STEM) fields.
The project repository will be stored on a publicly accessible server (https://github.com/hwsel). All the project data will be maintained for at least five years following the end of the grant period.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
This project developed a new content-based viewport prediction framework to reduce the bandwidth consumption when streaming live virtual reality (VR) videos from the content server to the end-user head mounted displays. During the course of the project, the PIs have accomplished three major research outcomes:
(1) a content-based viewport prediction framework for live VR video streaming considering the motion and semantics of the objects in the video;
(2) hardware/software techniques to enhance the speed and scalability of the proposed framework; and
(3) an evaluation framework to verify the functionality, performance, and scalability of the proposed viewport prediction framework.
Along with the research outcomes, the project has generated the following major products that address the intellectual merit and broader impacts of the conducted work.
Products that address intellectual merit:
(1) Publications: The project has generated research publications in top conferences/journals of multimedia, virtual reality, mobile computing, computer vision, machine learning signal processing, and computer architecture/hardware.
(2) Software releases: The project has generated several open source software releases for the viewport prediction framework, available online at https://github.com/hwsel/
(3) Dataset: The project has generated a 360-degree human activity video dataset, available online at https://egok360.github.io/
Products that address broader impacts:
(1) Education materials: The content and outcomes of the project have been incorporated as education materials in several undergraduate/graduate courses offered by the PIs in their specific institutions, such as Introduction to Machine Learning, Advanced Image Processing and Computer Vision, Deep Learning Embedded Systems, Advanced Topics on Computer Vision and Multimedia, and Advances in Deep Learning. The courses and education materials benefitted a large group of undergraduate and graduate students in their programs of studies and future career development.
(2) Student mentoring: The project has supported the PIs mentoring multiple graduate and undergraduate students in their specific institutions. The students have worked closely with the PIs in the planned research activities, with regularly scheduled research meetings and mentoring sessions. The students have learned the required knowledge and expertise in multimedia systems/signal processing, deep learning, and computer vision.
(3) Presentations and media coverages: The project has resulted in invited presentations at conferences, research institutions, and industry, as well as coverages at various media channels. The presentations and media coverages serve as important means of research dissemination to the society for potential technology transfer.
Last Modified: 11/29/2022
Modified by: Sheng Wei
Please report errors in award information by writing to: awardsearch@nsf.gov.
