Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so text-pipeline users (LLM agents, scrapers feeding markdown to other tools) don't need a separate HTML-to-markdown converter step. obscura fetch --dump markdown 'https://example.com' Layout: - crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a pub const so every layer can share the same DOM-walker. obscura-js is at the bottom of the dep DAG (obscura-browser -> obscura-js, obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single source of truth works without a circular dependency. - crates/obscura-browser/src/lib.rs re-exports it for ergonomics. - crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate and calls the shared const. - crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat variant + dump_markdown(&mut Page) which page.evaluate's the shared script and prints the result. Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs that's reborn as 3 lines, and ~10 lines added across the other four files. No behavior change for `--dump html|text|links`, and LP.getMarkdown over CDP keeps the exact same JSON shape. Four new unit tests in obscura-js cover headings, inline emphasis + links, ordered/unordered lists, and that <script> / <style> contents do not leak into the output. Generated by Ora Studio Vibe coded by ousamabenyounes Co-authored-by: Ora Studio <[email protected]>
This commit is contained in:
1 parent
9fbd0c73a3
commit
d8b7c5fad1
6 files changed
+150
-66
No files matched your search
@@ -109,6 +109,7 @@ enum DumpFormat {
|
||||
Html,
|
||||
Text,
|
||||
Links,
|
||||
Markdown,
|
||||
}
|
||||
|
||||
fn print_banner(port: u16) {
|
||||
@@ -395,6 +396,7 @@ async fn run_fetch(
|
||||
DumpFormat::Html => dump_html(&page),
|
||||
DumpFormat::Text => dump_text(&mut page),
|
||||
DumpFormat::Links => dump_links(&page),
|
||||
DumpFormat::Markdown => dump_markdown(&mut page),
|
||||
};
|
||||
write_or_print(rendered, output.as_ref()).await?;
|
||||
|
||||
@@ -454,6 +456,11 @@ fn dump_text(page: &mut Page) -> String {
|
||||
}).unwrap_or_default()
|
||||
}
|
||||
|
||||
fn dump_markdown(page: &mut Page) -> String {
|
||||
let result = page.evaluate(obscura_browser::HTML_TO_MARKDOWN_JS);
|
||||
result.as_str().unwrap_or_default().to_string()
|
||||
}
|
||||
|
||||
fn extract_readable_text(dom: &obscura_dom::DomTree, node_id: obscura_dom::NodeId) -> String {
|
||||
use obscura_dom::NodeData;
|
||||
|
||||
|
||||
Reference in new issue
Block a user