[FFmpeg-devel] [PATCH 00/24] Subtitle Filtering 2022
ffmpegagent
ffmpegagent at gmail.com
Fri Jan 14 03:13:09 EET 2022
Subtitle Filtering 2022
=======================
This is a substantial update to the earlier subtitle filtering patch series.
A primary goal has been to address others' concerns as much as possible on
one side and to provide more clarity and control over the way things are
working. Clarity is is specifically important to allow for a better
understanding of the need for a subtitle start pts value that can be
different from the frame's pts value. This is done by refactoring the
subtitle timing fields in AVFrame, adding a frame field to indicate repeated
subtitle frames, and finally the full removal of the heartbeat
functionality, replaced by a new 'subfeed' filter that provides different
modes for arbitrating subtitle frames in a filter graph. Finally, each
subtitle filter's documentation has been amended by a section describing the
filter's timeline behavior (in v2 update).
The update also includes major improvements to graphicsub2text and lots of
other details.
Versioning is restarting at v1 due to the new submission procedure.
All changes from previous version v24:
AVFrame
=======
* Removed sub_start_time The start time is now added to the subtitle
start_pts during decoding The sub_end_time field is adjusted accordingly
* Renamed sub_end_time to duration which it is effectively after removing
the start_time
* Added a sub-struct 'subtitle_timing' to av frame Contains subtitle_pts
renamed to 'subtitle_timing.start_pts' and 'subtitle_timing.duration'
* Change both fields to (fixed) time_base AV_TIMEBASE
* add repeat_sub field provides a clear indication whether a subtitle frame
is an actual subtitle event or a repeated subtitle frame in a filter
graph
Heartbeat Removal
=================
* completely removed the earlier heartbeat implementation
* filtering arbitration is now implemented in a new filter: 'subfeed'
* subfeed will be auto-inserted for compatiblity with sub2video command
lines
* the new behavior is not exactly identical to the earlier behavior, but it
basically allows to achieve the same results
New 'subfeed' Filter
====================
* a versatile filter for solving all kinds of problems with subtile frame
flow in filter graphs
* Can be inserted at any position in a graph
* Auto-inserted for sub2video command lines (in repeat-mode)
* Allows duration fixup delay input frames with unknown duration and infer
duration from start of subsequent frame
* Provides multiple modes of operation:
* repeat mode (default) Queues input frames Outputs frames at a fixed
(configurable) rate Either sends a matching input frame (repeatedly) or
empty frames otherwise
* scatter mode similar to repeat mode, but splits input frames by
duration into small segments with same content
* forward mode No fixed output rate Useful in combination with duration
fixup or overlap fixup
ffmpeg Tool Changes
===================
* delay subtitle output stream initialization (like for audio and video)
This is needed for example when a format header depends on having
received an initial frame to derive certain header values from
* decoding: set subtitle frame size from decoding context
* re-init graph when subtitle size changes
* always insert subscale filter for sub2video command lines (to ensure
correct scaling)
Subtitle Encoding
=================
* ignore repeated frames for encoding based on repeat_sub field in AVFrame
* support multi-area encoding for text subtitles Subtitle OCR can create
multiple areas at different positions. Previously, the texts were always
squashed into a single area ('subtitle rect'), which was not ideal.
Multiple text areas are now generally supported:
* ASS Encoder Changed to use the 'receive_packet' encoding API A single
frame with multiple text areas will create multiple packets now
* All other text subtitle encoders A newline is inserted between the text
from multiple areas
graphicsub2text (OCR)
=====================
* enhanced preprocessing
* using elbg algorithm for color quantization
* detection and removal of text outlines
* map-based identification of colors per word (text, outline, background)
* add option for duration fixup
* add option to dump preprocessing bitmaps
* Recognize formatting and apply as ASS inline styles
* per word(!)
* paragraph alignment
* positioning
* font names
* font size
* font style (italic, underline, bold)
* text color, outline color
Other Filter Changes
====================
* all: Make sure to forward all link properties (time base, frame rate, w,
h) where appropriate
* overlaytextsubs: request frames on the subtitle input
* overlaytextsubs: disable read-order checking
* overlaytextsubs: improve implementation of render_latest_only
* overlaytextsubs: ensure equal in/out video formats
* splitcc: derive framerate from realtime_latency
* graphicsub2video: implement caching of converted frames
* graphicsub2video: use 1x1 output frame size as long as subtitle size is
unknown (0x0)
Plus a dozen of things I forgot..
softworkz (24):
avcodec,avutil: Move enum AVSubtitleType to avutil, add new and
deprecate old values
avutil/frame: Prepare AVFrame for subtitle handling
avcodec/subtitles: Introduce new frame-based subtitle decoding API
avfilter/subtitles: Update vf_subtitles to use new decoding api
avcodec,avutil: Move ass helper functions to avutil as avpriv_ and
extend ass dialog parsing
avcodec/subtitles: Migrate subtitle encoders to frame-based API
avcodec/subtitles: Replace deprecated enum values
fftools/play,probe: Adjust for subtitle changes
avfilter/subtitles: Add subtitles.c for subtitle frame allocation
avfilter/avfilter: Handle subtitle frames
avfilter/avfilter: Fix hardcoded input index
avfilter/sbuffer: Add sbuffersrc and sbuffersink filters
avfilter/overlaygraphicsubs: Add overlaygraphicsubs and
graphicsub2video filters
avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video
filters
avfilter/textmod: Add textmod, censor and show_speaker filters
avfilter/stripstyles: Add stripstyles filter
avfilter/splitcc: Add splitcc filter for closed caption handling
avfilter/graphicsub2text: Add new graphicsub2text filter (OCR)
avfilter/subscale: Add filter for scaling and/or re-arranging
graphical subtitles
avfilter/subfeed: add subtitle feed filter
fftools/ffmpeg: Introduce subtitle filtering new frame-based subtitle
encoding
avutil/ass_split: Add parsing of hard-space tags (\h)
avcodec/webvttenc: convert hard-space tags to
doc/APIchanges: update for subtitle filtering changes
configure | 7 +-
doc/APIchanges | 24 +
doc/filters.texi | 756 ++++++++++
fftools/ffmpeg.c | 493 +++----
fftools/ffmpeg.h | 13 +-
fftools/ffmpeg_filter.c | 235 +++-
fftools/ffmpeg_hw.c | 2 +-
fftools/ffmpeg_opt.c | 3 +-
fftools/ffplay.c | 102 +-
fftools/ffprobe.c | 47 +-
libavcodec/Makefile | 56 +-
libavcodec/ass.h | 144 +-
libavcodec/assdec.c | 4 +-
libavcodec/assenc.c | 191 ++-
libavcodec/avcodec.h | 32 +-
libavcodec/ccaption_dec.c | 19 +-
libavcodec/codec_desc.c | 11 +
libavcodec/codec_desc.h | 8 +
libavcodec/decode.c | 56 +-
libavcodec/dvbsubdec.c | 2 +-
libavcodec/dvbsubenc.c | 96 +-
libavcodec/dvdsubdec.c | 2 +-
libavcodec/dvdsubenc.c | 102 +-
libavcodec/encode.c | 57 +-
libavcodec/internal.h | 22 +
libavcodec/jacosubdec.c | 2 +-
libavcodec/libaribb24.c | 2 +-
libavcodec/libzvbi-teletextdec.c | 14 +-
libavcodec/microdvddec.c | 7 +-
libavcodec/movtextdec.c | 3 +-
libavcodec/movtextenc.c | 126 +-
libavcodec/mpl2dec.c | 2 +-
libavcodec/pgssubdec.c | 2 +-
libavcodec/realtextdec.c | 2 +-
libavcodec/samidec.c | 2 +-
libavcodec/srtdec.c | 2 +-
libavcodec/srtenc.c | 116 +-
libavcodec/subviewerdec.c | 2 +-
libavcodec/tests/avcodec.c | 2 -
libavcodec/textdec.c | 4 +-
libavcodec/ttmlenc.c | 114 +-
libavcodec/utils.c | 184 +++
libavcodec/version.h | 2 +-
libavcodec/webvttdec.c | 2 +-
libavcodec/webvttenc.c | 100 +-
libavcodec/xsubdec.c | 2 +-
libavcodec/xsubenc.c | 88 +-
libavfilter/Makefile | 16 +
libavfilter/allfilters.c | 14 +
libavfilter/avfilter.c | 30 +-
libavfilter/avfilter.h | 11 +
libavfilter/avfiltergraph.c | 5 +
libavfilter/buffersink.c | 54 +
libavfilter/buffersink.h | 7 +
libavfilter/buffersrc.c | 72 +
libavfilter/buffersrc.h | 1 +
libavfilter/formats.c | 22 +
libavfilter/formats.h | 3 +
libavfilter/internal.h | 19 +-
libavfilter/sf_graphicsub2text.c | 1132 +++++++++++++++
libavfilter/sf_splitcc.c | 385 +++++
libavfilter/sf_stripstyles.c | 209 +++
libavfilter/sf_subfeed.c | 366 +++++
libavfilter/sf_subscale.c | 884 ++++++++++++
libavfilter/sf_textmod.c | 710 ++++++++++
libavfilter/subtitles.c | 63 +
libavfilter/subtitles.h | 44 +
libavfilter/vf_overlaygraphicsubs.c | 765 ++++++++++
libavfilter/vf_overlaytextsubs.c | 671 +++++++++
libavfilter/vf_subtitles.c | 56 +-
libavutil/Makefile | 4 +
{libavcodec => libavutil}/ass.c | 91 +-
libavutil/ass_internal.h | 135 ++
{libavcodec => libavutil}/ass_split.c | 37 +-
.../ass_split_internal.h | 32 +-
libavutil/frame.c | 211 ++-
libavutil/frame.h | 85 +-
libavutil/subfmt.c | 45 +
libavutil/subfmt.h | 115 ++
libavutil/version.h | 3 +-
tests/ref/fate/filter-overlay-dvdsub-2397 | 182 +--
tests/ref/fate/mov-mp4-ttml-dfxp | 8 +-
tests/ref/fate/mov-mp4-ttml-stpp | 8 +-
tests/ref/fate/sub-dvb | 162 ++-
tests/ref/fate/sub-textenc | 10 +-
tests/ref/fate/sub-ttmlenc | 8 +-
tests/ref/fate/sub-webvttenc | 10 +-
tests/ref/fate/sub2video | 1091 ++++++++++++++-
tests/ref/fate/sub2video_basic | 1239 +++++++++++++++--
tests/ref/fate/sub2video_time_limited | 78 +-
90 files changed, 10991 insertions(+), 1366 deletions(-)
create mode 100644 libavfilter/sf_graphicsub2text.c
create mode 100644 libavfilter/sf_splitcc.c
create mode 100644 libavfilter/sf_stripstyles.c
create mode 100644 libavfilter/sf_subfeed.c
create mode 100644 libavfilter/sf_subscale.c
create mode 100644 libavfilter/sf_textmod.c
create mode 100644 libavfilter/subtitles.c
create mode 100644 libavfilter/subtitles.h
create mode 100644 libavfilter/vf_overlaygraphicsubs.c
create mode 100644 libavfilter/vf_overlaytextsubs.c
rename {libavcodec => libavutil}/ass.c (65%)
create mode 100644 libavutil/ass_internal.h
rename {libavcodec => libavutil}/ass_split.c (93%)
rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%)
create mode 100644 libavutil/subfmt.c
create mode 100644 libavutil/subfmt.h
base-commit: c936c319bd54f097cc1d75b1ee1c407d53215d71
Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-18%2Fsoftworkz%2Fsubmit_subfiltering-v1
Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-18/softworkz/submit_subfiltering-v1
Pull-Request: https://github.com/ffstaging/FFmpeg/pull/18
--
ffmpeg-codebot
More information about the ffmpeg-devel
mailing list